RFC: Dependency Selector Syntax & npm query

accepted/0000-dependency-selector-syntax.md

@@ -0,0 +1,395 @@
+# `npm query` Command & Dependency Selector Syntax
+
+## Summary
+
+Introduce a new `npm query` commmand which exposes a new dependency selector syntax (informed by & respecting many aspects of the [CSS Selectors 4 Spec](https://dev.w3.org/csswg/selectors4/#relational)).
+
+## Motivation
+
+- Standardize the shape of, & querying of, dependency graphs with a robust object model, metadata & selector syntax
+- Leverage existing, known language syntax & operators from CSS to make disparate package information broadly accessible
+- Unlock the ability to answer complex, multi-faceted questions about dependencies, their relationships & associative metadata
+- Consolidate redundant logic of similar query commands in `npm` (ex. `npm fund`, `npm ls`, `npm outdated`, `npm audit` ...)
+
+## Detailed Explanation
+
+This RFC's spec & implementation should closely mimic the capabilities of existing CSS Selector specifications. Notably, we'll introduce limited net-new classes, states & syntax to ensure the widest adoption & understanding of paradigms. When deviating, we'll explicitely state why & how.
+
+## Implementation
+
+- `Arborist`'s `Node` Class will have a new `.querySelectorAll()` method
+  - this method will return a filtered, flattened dependency Arborist `Node` list based on a valid query selector
+- Introduce a new command, `npm query`, which will take a dependency selector & output a flattened dependency Node list (output is in `json` by default, but configurable)
+
+### Dependency Selector Syntax
+
+#### Overview:
+
+- there is no "type" or "tag" selectors (ex. `div, h1, a`) as a dependency/target is the only type of `Node` that can be queried
+- the term "dependencies" is in reference to any `Node` found in the `idealTree` returned by `Arborist`
+
+#### Selectors
+
+- `*` universal selector
+- `#<name>` dependency selector (equivalent to `[name="..."]`)
+- `#<name>@<version>` (equivalent to `[name=<name>]:semver(<version>)`)
+- `,` selector list delimiter
+- `.` class selector
+- `:` pseudo class selector
+- `>` direct decendent/child selector
+- `~` sibling selector
+
+#### Pseudo Selectors
+- [`:not(<selector>)`](https://developer.mozilla.org/en-US/docs/Web/CSS/:not)
+- [`:has(<selector>)`](https://developer.mozilla.org/en-US/docs/Web/CSS/:has)
+- [`:where(<selector list>)`](https://developer.mozilla.org/en-US/docs/Web/CSS/:where)
+- [`:is(<selector list>)`](https://developer.mozilla.org/en-US/docs/Web/CSS/:is)
+- [`:root`](https://developer.mozilla.org/en-US/docs/Web/CSS/:root) matches the root node/dependency
+- [`:scope`](https://developer.mozilla.org/en-US/docs/Web/CSS/:scope) matches node/dependency it was queried against
+- [`:empty`](https://developer.mozilla.org/en-US/docs/Web/CSS/:empty) when a dependency has no dependencies
+- [`:private`](https://docs.npmjs.com/cli/v8/configuring-npm/package-json#private) when a dependency is private
+- `:link` when a dependency is linked
+- `:deduped` when a dependency has been deduped
+- `:override` when a dependency is an override
+- `:extraneous` when a dependency exists but is not defined as a dependency of any node
+- `:invalid` when a dependency version is out of its ancestors specified range
+- `:missing` when a dependency is not found on disk
+- `:outdated` when a dependency is not `latest`
+- `:vulnerable` when a dependency has a `CVE` 
+- `:semver(<spec>)` matching a valid [`node-semver`](https://github.com/npm/node-semver) spec
+- `:path(<path>)` [glob](https://www.npmjs.com/package/glob) matching based on dependencies path relative to the project
+- `:realpath(<path>)` [glob](https://www.npmjs.com/package/glob) matching based on dependencies realpath
+- `:type(<type>)` [based on currently recognized types](https://github.com/npm/npm-package-arg#result-object)
+
+#### [Attribute Selectors](https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors)
+
+The attribute selector evaluates the key/value pairs in `package.json` if they are `String`s.
+
+- `[]` attribute selector (ie. existence of attribute)
+- `[attribute=value]` attribute value is equivalant...
+- `[attribute~=value]` attribute value contains word...
+- `[attribute*=value]` attribute value contains string...
+- `[attribute|=value]` attribute value is equal to or starts with...
+- `[attribute^=value]` attribute value begins with...
+- `[attribute$=value]` attribute value ends with...
+
+#### `Array` & `Object` Attribute Selectors
+
+The generic `:attr()` pseudo selector standardizes a pattern which can be used for attribute selection of `Object`s, `Array`s or `Arrays` of `Object`s accessible via `Arborist`'s `Node.package` metadata. This allows for iterative attribute selection beyond top-level `String` evaluation.
+
+`Array`s specifically use a special `item` keyword in place of a typical attribute name. `Arrays` also support exact `valu`e matching when a `String` is passed to the selector. See examples below:
+
+#### Example of an `Object`: 
+```css
+/* return dependencies that have a `scripts.test` containing `"tap"` */
+*:attr(:scripts([test~=tap]))
+```
+
+#### Example of an `Array` Attribute Selection:
+```css
+/* return dependencies that have a keyword that begins with "react" */
+*:attr(:keywords([item^="react"]))
+```
+
+#### Example of an `Array` matching directly to a value:
+```css
+/* return dependencies that have the exact keyword "react" */
+/* this is equivalent to `*:keywords([value="react"])` */
+*:attr(:keywords("react"))
+```
+
+#### Example of an `Array` of `Object`s: 
+```css
+/* returns */
+*:attr(:contributors([email="[email protected]"]))
+```
+
+### Classes
+
+Classes are defined by the package relationships to their ancestors (ie. the dependency types that are defined in `package.json`). This approach is user-centric as the ecosystem has been taught to think about dependencies in these classifications first-and-foremost. Dependencies are allowed to have multiple classifications (ex. a `workspace` may also be a `dev` dependency & may also be `bundled` - a selector for that type of dependency would look like: `*.workspace.dev.bundled`).
+
+- `.prod`
+- `.dev`
+- `.optional`
+- `.peer`
+- `.bundled`
+- `.workspace`
+
+### Command
+
+#### `npm query "<selector>"` (alias `q`)
+
+#### Options:
+
+- `query-output`
+  - Default: `json`
+  - Type: `json`, `list`, `explain`, `outdated`, `funding`, `audit`, `duplicates`, `file`
+
+#### Example Response Output
+
+- an array of dependency objects is returned which can contain multiple copies of the same package which may or may not have been linked or deduped
+
+```json
+[
+  {
+    "name": "",
+    "version": "",
+    "description": "",
+    "homepage": "",
+    "bugs": {},
+    "author": {},
+    "license": {},
+    "funding": {},
+    "files": [],
+    "main": "",
+    "browser": "",
+    "bin": {},
+    "man": [],
+    "directories": {},
+    "repository": {},
+    "scripts": {},
+    "config": {},
+    "dependencies": {},
+    "devDependencies": {},
+    "optionalDependencies": {},
+    "bundledDependencies": {},
+    "peerDependencies": {},
+    "peerDependenciesMeta": {},
+    "engines": {},
+    "os": [],
+    "cpu": [],
+    "workspaces": {},
+    "keywords": [],
+    "ancenstry": "",
+    "path": "",
+    "realpath": "",
+    "parent": "",
+    "vulnerabilities": [],
+    "cwe": []
+    ...
+  },
+  …
+```
+
+#### Usage
+
+```bash
+npm query ":root > .workspace > *"  # get all workspace direct deps
+```
+
+### Extended Example Queries & Use Cases
+
+```stylus
+// all deps
+*
+
+// all direct deps
+:root > *
+
+// direct production deps
+:root > .prod
+
+// direct development deps
+:root > .dev 
+
+// any peer dep of a direct deps
+:root > * > .peer  
+
+// any workspace dep
+.workspace
+
+// all workspaces that depend on another workspace
+.workspace > .workspace
+
+// all workspaces that have peer deps
+.workspace:has(*.peer)
+
+// any dep named "lodash"
+// equivalent to [name="lodash"]
+#lodash 
+
+// any deps named "lodash" & within semver range ^"1.2.3"
+#lodash@^1.2.3
+// equivalent to...
+[name="lodash"]:semver(^1.2.3)
+
+// get the hoisted node for a given semver range
+#lodash@^1.2.3:not(:deduped)
+
+// querying deps with a specific version
+#[email protected]
+// equivalent to...
+[name="lodash"][version="2.1.5"]
+
+// all deps living alongside vulnerable deps
+*:vulnerable ~ *:not(:vulnerable)
+
+// has any deps
+*:has(*)
+
+// has any vulnerable deps
+*:has(*:vulnerable)
+
+// deps with no other deps (ie. "leaf" nodes)
+*:empty
+
+// all vulnerable deps that aren't dev deps & that aren't vulnerable to CWE-1333
+*:vulnerable:not(.dev:cwe(1333))
+
+// manually querying git dependencies
+*[repository^="github:"],
+*[repository^="git:"],
+*[repository^="https://github.com"],
+*[repository^="http://github.com"],
+*[repository^="https://github.com"],
+*[repository^="+git:..."]
+
+// querying for all git dependencies
+*:type(git)
+
+// find all references to "install" scripts
+*[scripts=install], 
+*[scripts=postinstall],
+*[scripts=preinstall]
+
+// get production dependencies that aren't also dev deps
+.prod:not(.dev)
+
+// get dependencies with specific licenses
+*[license="MIT"], *[license="ISC"]
+
+// find all packages that have @ruyadorno as a contributor
+*:attr(:contributors([[email protected]]))
+```
+
+## Next Steps
+
+### Command Mapping to `query-output`
+
+Previous commands with similar behaivours will now be able to utilize `Aborist` `Node.querySelectorAll()` under-the-hood & will fast-follow the `npm query` implementation.
+
+#### `npm list`
+
+```bash
+npm list # equivalent to...
+npm query ":root > *"
+
+npm list --all # equivalent to...
+npm query "*" --query-output list
+
+npm list <pkg> # equivalent to...
+npm query "#<pkg>" --query-output list
+```
+
+#### `npm explain`
+
+```bash
+npm explain <pkg> # equivalent to...
+npm query "#<pkg>" --query-output explain
+```
+
+#### `npm outdated`
+
+```bash
+npm outdated # equivalent to...
+npm query ":root > *:outdated" --query-output outdated
+
+npm outdated --all # equivalent to...
+npm query "*:outdated" --query-output outdated
+```
+
+#### `npm fund`
+
+```bash
+npm fund # equivalent to...
+npm query ":root > *[funding]" --query-output funding
+
+npm fund --all # equivalent to...
+npm query "*[funding]" --query-output funding
+
+npm fund <pkg> # equivalent to...
+npm query "#<pkg>" --query-output funding
+```
+
+#### `npm audit`
+
+```bash
+npm audit # equivalent to...
+npm query ":root > *:vulnerable" --query-output audit
+
+npm audit --all # equivalent to...
+npm query "*:vulnerable" --query-output audit
+
+npm audit <pkg> # equivalent to...
+npm query "#<pkg>" --query-output audit
+```
+
+#### `npm find-dupes`
+
+```bash
+npm find-dupes # equivalent to...
+npm query "*:deduped" --query-output duplicates
+```
+
+#### `npm view`
+
+```bash
+npm view # equivalent to...
+npm query ":root" --query-output view
+
+npm view <pkg> # equivalent to...
+npm query "#<pkg>" --query-output view
+```
+
+## Commands _could_ read from `stdin`
+
+In a future RFC, & major version bump, `npm` could begin reading from `stdin` to chain commands together with a common understand of a dependency object. All of the below commands would add the ability to execute as if they were passed package specs (our current defulat representation of packages/dependencies).
+
+```
+audit, bugs, ci, config, deprecate, diff, dist-tag, docs, edit, exec, 
+explain, explore, find-dupes, fund, get, install, install-ci-test, 
+install-test, link, list, outdated, pack, pkg, prune, publish, rebuild, 
+repo, restart, run-script, set, set-script, shrinkwrap, star, stars,
+start, stop, team, test, token, uninstall, unpublish, unstar, update,
+version, view
+```
+
+### Example of piping from `npm query` to other commands
+
+```bash
+# list workspaces w/ peer deps
+npm query ".workspace:has(.peer)" | npm ls
+
+# list outdated direct dev deps
+npm query ":root > .dev:outdated" | npm outdated
+
+# install the same dev deps across all workspaces
+npm query ":root > .dev" | npm install --workspaces
+
+# show audit details for dependencies with a specific vulnerability/CWE
+npm query "*:cwe(1333)" | npm audit
+
+# show audit details for vulnerable deps that aren't ReDoS dev deps
+npm query "*:vulnerable:not(.dev:cwe(1333))" | npm audit
+```
+
+## Prior Art
+
+- [`HTML`](https://html.spec.whatwg.org/) & [DOM](https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model) Specifications
+- [`CSS`](https://www.w3.org/Style/CSS/specs.en.html), [Selectors](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors) & [Pseudo Class](https://developer.mozilla.org/en-US/docs/Web/CSS/Pseudo-classes) Specifications
+- AST Selector Libraries/Parsers
+  - [`estree`](https://github.com/estree/estree)
+  - [`abstract-syntax-tree`](https://www.npmjs.com/package/abstract-syntax-tree)
+  - [`postcss-selector-parser`](https://www.npmjs.com/package/postcss-selector-parser) / [API](https://github.com/postcss/postcss-selector-parser/blob/master/API.md) 
+  - [`css-selector-parser`](https://www.npmjs.com/package/css-selector-parser)
+- [pnpm's `--filter`](https://pnpm.io/filtering) Flag
+- [Gzemnid](https://github.com/nodejs/Gzemnid) Package Querying
+
+## F.A.Q.
+- Q. Is there any such thing as a bare specifier?
+  - A. No. Unlike CSS Selectors, there's no `"element"`, or `"dependency"` in this context, equivalent. The selector syntax presuposes all entities are packages.
+- Q. Should this syntax cover **all** possible queries of a dependency graph?
+  - A. No. This spec is meant to provide a sufficently mature mechanism/building block for answering the majority of questions end-users have about their depndencies (re. 80/20 rule applies here)
+
+## Unresolved Questions & Bikeshedding
+- N/A