lib: create diagnostics_channel module #34895

Qard · 2020-08-23T20:13:02Z

I've continued the effort started in nodejs/diagnostics#180 to create an API focused on high-performance reporting of arbitrary diagnostics data via named channels. The intent is to layer some more complex concepts on top of this event stream to produce things like metrics or spans for tracing. This is drawing on some ideas from a previous experiment of mine to build a generic data pipeline abstracted by more structural constructs here: https://github.com/Qard/universal-monitoring-bus/blob/master/example.js

This is still somewhat a work-in-progress, but is usable in its current state and should already perform fairly well. It is carefully optimized to avoid lookups and to pull expensive behaviours as much as possible out of the publish path and put the expense exclusively on subscription and channel creation. There is some duplication in the API as it is currently as I want to present this to the community and gather feedback on what parts should stay, what parts should go, and also if there are any things that should be added or changed.

Also, the reason for making this a core module and not just something on npm is that the intent is largely to use this to gather diagnostics data from within Node.js core itself, which requires that it exist within Node.js. It is designed as a public API so that some other userland libraries may also make use of it to report diagnostic data. The hope is for an API such as this to largely eliminate the need for APM vendors to patch Node.js core, the module loader and much of the external module ecosystem, which is somewhat fragile and has a significant performance impact.

Please keep in mind:

APM vendors very much need a solution to eliminate the need for extensive patching
This is very much an experiment and is likely to evolve significantly before considering actually landing it
I understand there is likely to be pushback to this for various reasons. I'll be working to alleviate any concerns that are shared here. If that requires significant changes, so be it.

cc @nodejs/diagnostics

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

bmeck · 2020-08-23T21:31:38Z

It would be good to give example reasons on when/why to use these APIs rather than what these APIs do.

lib/diagnostics_channel.js

Qard · 2020-08-23T21:39:51Z

Yep, planning on expanding docs further. I just put together the current docs today as a quick intro to how it works. I'll continue to build out the docs further as API details are solidified more and we build a clearer idea of the higher level constructs to layer over top of it. I plan to discuss this at the next diagnostics meeting to figure out if this is a sufficient code to the higher-level APIs we're thinking of.

lib/diagnostics_channel.js

mcollina

Have you researched approaches based on a trie, e.g. https://www.npmjs.com/package/qlobber?

doc/api/diagnostics_channel.md

lib/diagnostics_channel.js

doc/api/diagnostics_channel.md

lib/diagnostics_channel.js

Qard · 2020-08-24T06:29:01Z

No, I have not really looked into possible trie uses. I figured more complicated matching could just be implemented on top of the shouldListen function. I expect the majority of cases would be that one subscribing to events would listen to specific channels and that more complicated matching would be somewhat of a niche need--I'm not even 100% certain it's needed.

mcollina · 2020-08-24T07:30:02Z

The advantage of a trie data structure is to avoid the memory leak this implementation has.

doc/api/diagnostics_channel.md

Notable changes: child_process: * (SEMVER-MINOR) add ChildProcess 'spawn' event (Matthew Francis Brunetti) #35369 dns: * (SEMVER-MINOR) add setLocalAddress to Resolver (Josh Dague) #34824 http: * (SEMVER-MINOR) report request start and end with diagnostics_channel (Stephen Belanger) #34895 http2: * (SEMVER-MINOR) add updateSettings to both http2 servers (Vincent Boivin) #35383 lib: * (SEMVER-MINOR) create diagnostics_channel module (Stephen Belanger) #34895 src: * (SEMVER-MINOR) add --heapsnapshot-near-heap-limit option (Joyee Cheung) #33010 v8: * (SEMVER-MINOR) implement v8.stopCoverage() (Joyee Cheung) #33807 * (SEMVER-MINOR) implement v8.takeCoverage() (Joyee Cheung) #33807 worker: * (SEMVER-MINOR) add eventLoopUtilization() (Trevor Norris) #35664 PR-URL: #35948

PR-URL: #34895 Reviewed-By: Bryan English <[email protected]> Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: Vladimir de Turckheim <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Gabriel Schulhof <[email protected]> Reviewed-By: Michael Dawson <[email protected]>

Notable changes: * assert: * change status of legacy asserts (James M Snell) (#38113) * doc: * apply style for legacy status (James M Snell) (#37784) * revoke deprecation of legacy url, change status to legacy (James M Snell) (#37784) * add legacy status to stability index (James M Snell) (#37784) * upgrade stability status of report API (Gireesh Punathil) (#35654) * deps: * V8: cherry-pick cf71540c684c (BoHong Li) (#38051) * update ICU to 68.1 (Michaël Zasso) (#36187) * upgrade to libuv 1.41.0 (Colin Ihrig) (#37360) * fs: * add promisified readFile benchmark (Nitzan Uziely) (#37608) * http: * add http.ClientRequest.getRawHeaderNames() (simov) (#37660) * report request start and end with diagnostics\_channel (Stephen Belanger) (#34895) * lib: * create diagnostics\_channel module (Stephen Belanger) (#34895) * implement AbortSignal.abort() (James M Snell) (#37693) * test: * update dom/abort tests (James M Snell) (#37693) * util: * add getSystemErrorMap() impl (eladkeyshawn) (#38101)

PR-URL: #38507 Notable changes: * assert: * change status of legacy asserts (James M Snell) (#38113) * doc: * apply style for legacy status (James M Snell) (#37784) * revoke deprecation of legacy url, change status to legacy (James M Snell) (#37784) * add legacy status to stability index (James M Snell) (#37784) * upgrade stability status of report API (Gireesh Punathil) (#35654) * deps: * V8: cherry-pick cf71540c684c (BoHong Li) (#38051) * update ICU to 68.1 (Michaël Zasso) (#36187) * upgrade to libuv 1.41.0 (Colin Ihrig) (#37360) * fs: * add promisified readFile benchmark (Nitzan Uziely) (#37608) * http: * add http.ClientRequest.getRawHeaderNames() (simov) (#37660) * report request start and end with diagnostics\_channel (Stephen Belanger) (#34895) * lib: * create diagnostics\_channel module (Stephen Belanger) (#34895) * implement AbortSignal.abort() (James M Snell) (#37693) * test: * update dom/abort tests (James M Snell) (#37693) * util: * add getSystemErrorMap() impl (eladkeyshawn) (#38101)

PR-URL: #38507 Notable changes: * assert: * change status of legacy asserts (James M Snell) (#38113) * crypto: * implement randomuuid (James M Snell) (#36729) * doc: * apply style for legacy status (James M Snell) (#37784) * revoke deprecation of legacy url, change status to legacy (James M Snell) (#37784) * add legacy status to stability index (James M Snell) (#37784) * upgrade stability status of report API (Gireesh Punathil) (#35654) * deps: * V8: Backport various patches for Apple Silicon support (BoHong Li) (#38051) * update ICU to 68.1 (Michaël Zasso) (#36187) * upgrade to libuv 1.41.0 (Colin Ihrig) (#37360) * http: * add http.ClientRequest.getRawHeaderNames() (simov) (#37660) * report request start and end with diagnostics\_channel (Stephen Belanger) (#34895) * lib: * create diagnostics\_channel module (Stephen Belanger) (#34895) * implement AbortSignal.abort() (James M Snell) (#37693) * test: * update dom/abort tests (James M Snell) (#37693) * util: * add getSystemErrorMap() impl (eladkeyshawn) (#38101)

PR-URL: #38507 Notable changes: * crypto: * implement randomuuid (James M Snell) (#36729) * doc: * apply style for legacy status (James M Snell) (#37784) * revoke deprecation of legacy url, change status to legacy (James M Snell) (#37784) * add legacy status to stability index (James M Snell) (#37784) * upgrade stability status of report API (Gireesh Punathil) (#35654) * deps: * V8: Backport various patches for Apple Silicon support (BoHong Li) (#38051) * update ICU to 68.1 (Michaël Zasso) (#36187) * upgrade to libuv 1.41.0 (Colin Ihrig) (#37360) * http: * add http.ClientRequest.getRawHeaderNames() (simov) (#37660) * report request start and end with diagnostics\_channel (Stephen Belanger) (#34895) * lib: * create diagnostics\_channel module (Stephen Belanger) (#34895) * implement AbortSignal.abort() (James M Snell) (#37693) * test: * update dom/abort tests (James M Snell) (#37693) * util: * add getSystemErrorMap() impl (eladkeyshawn) (#38101)

Notable changes: Diagnostics channel (experimental module): `diagnostics_channel` is a new experimental module that provides an API to create named channels to report arbitrary message data for diagnostics purposes. The module was initially introduced in Node.js v15.1.0 and is backported to v14.17.0 to enable testing it at a larger scale. With `diagnostics_channel`, Node.js core and module authors can publish contextual data about what they are doing at a given time. This could be the hostname and query string of a mysql query, for example. Just create a named channel with `dc.channel(name)` and call `channel.publish(data)` to send the data to any listeners to that channel. ```js const dc = require('diagnostics_channel'); const channel = dc.channel('mysql.query'); MySQL.prototype.query = function query(queryString, values, callback) { // Broadcast query information whenever a query is made channel.publish({ query: queryString, host: this.hostname, }); this.doQuery(queryString, values, callback); }; ``` Channels are like one big global event emitter but are split into separate objects to ensure they get the best performance. If nothing is listening to the channel, the publishing overhead should be as close to zero as possible. Consuming channel data is as easy as using `channel.subscribe(listener)` to run a function whenever a message is published to that channel. ```js const dc = require('diagnostics_channel'); const channel = dc.channel('mysql.query'); channel.subscribe(({ query, host }) => { console.log(`mysql query to ${host}: ${query}`); }); ``` The data captured can be used to provide context for what an app is doing at a given time. This can be used for things like augmenting tracing data, tracking network and filesystem activity, logging queries, and many other things. It's also a very useful data source for diagnostics tools to provide a clearer picture of exactly what the application is doing at a given point in the data they are presenting. Contributed by Stephen Belanger (#34895). UUID support in the crypto module: The new `crypto.randomUUID()` method now allows to generate random [RFC 4122](https://www.rfc-editor.org/rfc/rfc4122.txt) Version 4 UUID strings: ```js const { randomUUID } = require('crypto'); console.log(randomUUID()); // 'aa7c91a1-f8fc-4339-b9db-f93fc7233429' ``` Contributed by James M Snell (#36729. Experimental support for `AbortController` and `AbortSignal`: Node.js 14.17.0 adds experimental partial support for `AbortController` and `AbortSignal`. Both constructors can be enabled globally using the `--experimental-abortcontroller` flag. Additionally, several Node.js APIs have been updated to support `AbortSignal` for cancellation. It is not mandatory to use the built-in constructors with them. Any spec-compliant third-party alternatives should be compatible. `AbortSignal` support was added to the following methods: * `child_process.exec` * `child_process.execFile` * `child_process.fork` * `child_process.spawn` * `dgram.createSocket` * `events.on` * `events.once` * `fs.readFile` * `fs.watch` * `fs.writeFile` * `http.request` * `https.request` * `http2Session.request` * The promisified variants of `setImmediate` and `setTimeout` Other notable changes: * doc: * revoke deprecation of legacy url, change status to legacy (James M Snell) (#37784) * add legacy status to stability index (James M Snell) (#37784) * upgrade stability status of report API (Gireesh Punathil) (#35654) * deps: * V8: Backport various patches for Apple Silicon support (BoHong Li) (#38051) * update ICU to 68.1 (Michaël Zasso) (#36187) * upgrade to libuv 1.41.0 (Colin Ihrig) (#37360) * http: * add http.ClientRequest.getRawHeaderNames() (simov) (#37660) * report request start and end with diagnostics\_channel (Stephen Belanger) (#34895) * util: * add getSystemErrorMap() impl (eladkeyshawn) (#38101) PR-URL: #38507

Notable Changes: Diagnostics channel (experimental module): `diagnostics_channel` is a new experimental module that provides an API to create named channels to report arbitrary message data for diagnostics purposes. The module was initially introduced in Node.js v15.1.0 and is backported to v14.17.0 to enable testing it at a larger scale. With `diagnostics_channel`, Node.js core and module authors can publish contextual data about what they are doing at a given time. This could be the hostname and query string of a mysql query, for example. Just create a named channel with `dc.channel(name)` and call `channel.publish(data)` to send the data to any listeners to that channel. ```js const dc = require('diagnostics_channel'); const channel = dc.channel('mysql.query'); MySQL.prototype.query = function query(queryString, values, callback) { // Broadcast query information whenever a query is made channel.publish({ query: queryString, host: this.hostname, }); this.doQuery(queryString, values, callback); }; ``` Channels are like one big global event emitter but are split into separate objects to ensure they get the best performance. If nothing is listening to the channel, the publishing overhead should be as close to zero as possible. Consuming channel data is as easy as using `channel.subscribe(listener)` to run a function whenever a message is published to that channel. ```js const dc = require('diagnostics_channel'); const channel = dc.channel('mysql.query'); channel.subscribe(({ query, host }) => { console.log(`mysql query to ${host}: ${query}`); }); ``` The data captured can be used to provide context for what an app is doing at a given time. This can be used for things like augmenting tracing data, tracking network and filesystem activity, logging queries, and many other things. It's also a very useful data source for diagnostics tools to provide a clearer picture of exactly what the application is doing at a given point in the data they are presenting. Contributed by Stephen Belanger (#34895). UUID support in the crypto module: The new `crypto.randomUUID()` method now allows to generate random [RFC 4122](https://www.rfc-editor.org/rfc/rfc4122.txt) Version 4 UUID strings: ```js const { randomUUID } = require('crypto'); console.log(randomUUID()); // 'aa7c91a1-f8fc-4339-b9db-f93fc7233429' ``` Contributed by James M Snell (#36729). Experimental support for `AbortController` and `AbortSignal`: Node.js 14.17.0 adds experimental partial support for `AbortController` and `AbortSignal`. Both constructors can be enabled globally using the `--experimental-abortcontroller` flag. Additionally, several Node.js APIs have been updated to support `AbortSignal` for cancellation. It is not mandatory to use the built-in constructors with them. Any spec-compliant third-party alternatives should be compatible. `AbortSignal` support was added to the following methods: * `child_process.exec` * `child_process.execFile` * `child_process.fork` * `child_process.spawn` * `dgram.createSocket` * `events.on` * `events.once` * `fs.readFile` * `fs.watch` * `fs.writeFile` * `http.request` * `https.request` * `http2Session.request` * The promisified variants of `setImmediate` and `setTimeout` Other notable changes: * doc: * revoke deprecation of legacy url, change status to legacy (James M Snell) (#37784) * add legacy status to stability index (James M Snell) (#37784) * upgrade stability status of report API (Gireesh Punathil) (#35654) * deps: * V8: Backport various patches for Apple Silicon support (BoHong Li) (#38051) * update ICU to 68.1 (Michaël Zasso) (#36187) * upgrade to libuv 1.41.0 (Colin Ihrig) (#37360) * http: * add http.ClientRequest.getRawHeaderNames() (simov) (#37660) * report request start and end with diagnostics\_channel (Stephen Belanger) (#34895) * util: * add getSystemErrorMap() impl (eladkeyshawn) (#38101) PR-URL: #38507

Refs: - nodejs/node#34186 - nodejs/node#34895 - nodejs/node#33527 - nodejs/node#33950

Qard added lib / src Issues and PRs related to general changes in the lib or src directory. diag-agenda Issues and PRs to discuss during the meetings of the diagnostics working group. labels Aug 23, 2020

Qard requested review from mcollina and mmarchini August 23, 2020 20:13

nodejs-github-bot added the build Issues and PRs related to build files or the CI. label Aug 23, 2020

Qard mentioned this pull request Aug 23, 2020

Diagnostics Channel - Next steps nodejs/diagnostics#180

Closed

8 tasks

Qard requested a review from Flarna August 23, 2020 20:16

Qard force-pushed the diagnostics-channel branch 4 times, most recently from 6e2b771 to c9c3787 Compare August 23, 2020 21:28

bmeck reviewed Aug 23, 2020

View reviewed changes