src,doc: add C++ internals documentation #30552

addaleax · 2019-11-20T00:04:14Z

This aims to help explain some of the internal patterns and utilities
that we use. It is by no means exhaustive, and suggestions for
additions are welcome.

Some of this is based on the existing work from #26929.

Refs: #26929

/cc @vsemozhetbyt @jasnell @sam-github

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
documentation is changed or added
commit message follows commit guidelines

This aims to help explain some of the internal patterns and utilities that we use. It is by no means exhaustive, and suggestions for additions are welcome. Some of this is based on the existing work from nodejs#26929. Refs: nodejs#26929

sam-github

Nice!

src/README.md

gengjiawen

I am strongly inclined to add link to https://github.com/nodejs/node/blob/master/doc/api/addons.md. Also maybe put it docs ?

src/README.md

Co-Authored-By: mscdex <[email protected]>

Co-Authored-By: Jiawen Geng <[email protected]>

addaleax · 2019-11-20T11:11:35Z

I am strongly inclined to add link to https://github.com/nodejs/node/blob/master/doc/api/addons.md.

The addon docs aren’t good for this, imo. The parts about describing how to set up addons are not relevant here (e.g. NODE_MODULE_... macros) and the examples either don’t apply or don’t reflect best practices.

Also maybe put it docs ?

If you mean moving it to doc/, eh, yeah, I guess we could do that but I like having internals docs close to the source code. (i.e. I’d prefer src/doc/ over something in doc/ but since we don’t have any other C++ docs I’d say src/README.md is fine).

src/README.md

gengjiawen · 2019-11-20T11:30:50Z

The addon docs aren’t good for this, imo. The parts about describing how to set up addons are not relevant here (e.g. NODE_MODULE_... macros) and the examples either don’t apply or don’t reflect best practices.

Addon can use v8 api, node.h things like that. I think current doc lack this kind of content. This will give developers many help if we include this.

If you mean moving it to doc/, eh, yeah, I guess we could do that but I like having internals docs close to the source code.

To be fare, they are still in the same repo if we put it in doc. I do think this kind of content need more exposure. It can be very helpful and valuable to whole community.

addaleax · 2019-11-20T11:39:11Z

@gengjiawen @gireeshpunathil PTAL

addaleax · 2019-11-20T11:43:40Z

The addon docs aren’t good for this, imo. The parts about describing how to set up addons are not relevant here (e.g. NODE_MODULE_... macros) and the examples either don’t apply or don’t reflect best practices.

Addon can use v8 api, node.h things like that. I think current doc lack this kind of content. This will give developers many help if we include this.

Yeah, and if we had good addon docs, that would be massively helpful and I’d totally be in favour of linking them- But imo they’re not helpful in their current state which is kind of okay for me given that we’ve been pushing N-API for a while.

If you mean moving it to doc/, eh, yeah, I guess we could do that but I like having internals docs close to the source code.

To be fare, they are still in the same repo if we put it in doc. I do think this kind of content need more exposure. It can be very helpful and valuable to whole community.

I mean, these are docs about internals specifically. I sure hope that they are helpful and valuable to people who work on files in src/, and so having the visibility for those people has the highest priority for me.

joyeecheung

I have some comments, but nothing blocking. Thanks for working on this!

src/README.md

joyeecheung · 2019-11-20T12:21:09Z

src/README.md

+
+There is no way to abort libuv requests in general. If a libuv request is not
+managed through a [`ReqWrap`][] instance, the
+`env->IncreaseWaitingRequestCounter()` and


I am slightly concerned direct references to these methods could get out of date. Can't we just write these down in env.h and link to the headers instead?

Can't we just write these down in env.h and link to the headers instead?

I’m not sure what you mean? But either way, I think keeping this doc up to date and making sure of that during reviews is something we’ll have to do either way.

I think keeping this doc up to date and making sure of that during reviews is something we’ll have to do either way.

I think if keeping this doc up-to-date is going to be a blocker for landing changes in src/, we should just move the information about the internal classes into the headers as comments, and link to those in the README. For one if the README is too long it becomes difficult to read, also it's going to be a pain to remember updating this doc while changing the code - but slightly less so if the comments live in the headers.

Also it feels weird to me that in order to understand some of these classes at a higher level, you either have to read the code, or read this doc because what's supposed to be said in comments are moved into a markdown file...I think it would be more helpful if the README is just a table of contents.

tbh, I really like the proposed format. It has a flow, the necessary level of abstraction, example code at right places, and caveats and cautions as appropriate - something either not possible in code comments, or not easily consumable. IMO inline comments are rich with the context, but weak in connecting bigger views, and more importantly, discoverability.

If you have items that you feel should also be documented more explicitly in a header file, I’m happy to do that, but generally I agree with @gireeshpunathil – this is not supposed to be an exhaustive guide or reference documentation, but rather paint the big picture and distinguish what’s important from what’s not.

and more importantly, discoverability.

I feel the opposite - I think content in the README is more difficult to discover than content in the headers, especially when the README becomes long. For me it's rare that I'll read a doc, top to bottom, to understand a code base and remember all these identifier names. I usually run into these names when I hack on the code and I expect to see explanations in the headers.

this is not supposed to be an exhaustive guide or reference documentation, but rather paint the big picture and distinguish what’s important from what’s not.

That's what I think this document should be like as well, but with details like

They can be added and removed through by using env->AddCleanupHook(callback, hint); and env->RemoveCleanupHook(callback, hint);, where callback takes a void* hint argument.

and

If a libuv request is not managed through a [ReqWrap][] instance, the env->IncreaseWaitingRequestCounter() and env->DecreaseWaitingRequestCounter() functions need to be used to keep track of the number of active libuv requests.

It does not seem to be the case. These references to specific method names and their arguments do not seem to be necessary in terms of giving the reader a bigger view of the code base.

and more importantly, discoverability.

I feel the opposite - I think content in the README is more difficult to discover than content in the headers, especially when the README becomes long.

That’s somewhat solvable by adding a table of contents, if you like?

this is not supposed to be an exhaustive guide or reference documentation, but rather paint the big picture and distinguish what’s important from what’s not.

That's what I think this document should be like as well, but with details like

They can be added and removed through by using env->AddCleanupHook(callback, hint); and env->RemoveCleanupHook(callback, hint);, where callback takes a void* hint argument.

and

If a libuv request is not managed through a [ReqWrap][] instance, the env->IncreaseWaitingRequestCounter() and env->DecreaseWaitingRequestCounter() functions need to be used to keep track of the number of active libuv requests.

It does not seem to be the case. These references to specific method names and their arguments do not seem to be necessary in terms of giving the reader a bigger view of the code base.

Well … where would you document these, especially the latter one? It’s easy to add inline docs to ReqWrap but what do you do when you want to describe when it’s about not using ReqWrap?

In any case, I don’t feel like these give away too much detail. Judging from the current contents of the file in this PR, the number of yearly PRs that would require updating it is probably in the single digits.

That’s somewhat solvable by adding a table of contents, if you like?

I think that does not help in this case. To quote my edit:

For me it's rare that I'll read a doc, top to bottom, to understand a code base and remember all these identifier names. I usually run into these names when I hack on the code and I expect to see explanations in the headers.

Now this is solvable if in the headers we point the readers to the README for more information, but that feels kind of weird.

where would you document these, especially the latter one?

I think if the goal is to give a big picture in the README, for the latter one it can just say something like:

If a libuv request is not managed through a [ReqWrap][] instance, a counter is used to make sure it gets cleaned up.

And the method names can be documented in the req_wrap.h or env.h. The README can just point the readers to that header to see these details. This is a README for src/ after all, so I'd expect it to be more of a table of contents for all these files under src/.

src/README.md

Co-Authored-By: Colin Ihrig <[email protected]>

addaleax · 2019-11-20T16:25:02Z

@cjihrig Done, and added documentation for internal fields; PTAL

src/README.md

gireeshpunathil · 2019-11-22T07:25:25Z

Ci-lite: https://ci.nodejs.org/job/node-test-pull-request-lite-pipeline/4081

lundibundi

LGTM, great work.

src/README.md

jasnell

WOO! 🎉 💯 🥇

Co-Authored-By: Denys Otrishko <[email protected]>

mhdawson

LGTM, @addaleax thanks for putting it together. Great resource for new contributors.

This aims to help explain some of the internal patterns and utilities that we use. It is by no means exhaustive, and suggestions for additions are welcome. Some of this is based on the existing work from #26929. Refs: #26929 PR-URL: #30552 Reviewed-By: Sam Roberts <[email protected]> Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Jiawen Geng <[email protected]> Reviewed-By: David Carlier <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]> Reviewed-By: Denys Otrishko <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Michael Dawson <[email protected]>

addaleax · 2019-11-22T20:28:53Z

Landed in c132035, thanks for the reviews!

vsemozhetbyt · 2019-11-24T10:59:38Z

src/README.md

+[`async_wrap.h`]: async_wrap.h
+[`base_object.h`]: base_object.h
+[`handle_wrap.h`]: handle_wrap.h
+[`memory_retainer.h`]: memory_retainer.h


Sorry to be late. This link seems 404 now.

vsemozhetbyt · 2019-11-24T11:06:52Z

src/README.md

+[cleanup hooks]: #cleanup-hooks
+[event loop]: #event-loop
+[exception handling]: #exception-handling
+[internal field]: #internal-field


- [internal field]: #internal-field + [internal field]: #internal-fields

vsemozhetbyt · 2019-11-24T15:41:18Z

src/README.md

+data if it is typically expected to be small (e.g. file paths).
+
+The `Utf8Value`, `TwoByteValue` (i.e. UTF-16 value) and `BufferValue`
+(`Utf8Value` but copy data from a `Buffer` is that is passed) helpers


Is there a typo here?

Refs: nodejs#30552 (review)

Refs: #30552 (review) PR-URL: #30693 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Jiawen Geng <[email protected]> Reviewed-By: Trivikram Kamat <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

This aims to help explain some of the internal patterns and utilities that we use. It is by no means exhaustive, and suggestions for additions are welcome. Some of this is based on the existing work from #26929. Refs: #26929 PR-URL: #30552 Reviewed-By: Sam Roberts <[email protected]> Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Jiawen Geng <[email protected]> Reviewed-By: David Carlier <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]> Reviewed-By: Denys Otrishko <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Michael Dawson <[email protected]>

Refs: #30552 (review) PR-URL: #30693 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Jiawen Geng <[email protected]> Reviewed-By: Trivikram Kamat <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

This aims to help explain some of the internal patterns and utilities that we use. It is by no means exhaustive, and suggestions for additions are welcome. Some of this is based on the existing work from #26929. Refs: #26929 PR-URL: #30552 Reviewed-By: Sam Roberts <[email protected]> Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Jiawen Geng <[email protected]> Reviewed-By: David Carlier <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]> Reviewed-By: Denys Otrishko <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Michael Dawson <[email protected]>

Refs: #30552 (review) PR-URL: #30693 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Jiawen Geng <[email protected]> Reviewed-By: Trivikram Kamat <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

This aims to help explain some of the internal patterns and utilities that we use. It is by no means exhaustive, and suggestions for additions are welcome. Some of this is based on the existing work from #26929. Refs: #26929 PR-URL: #30552 Reviewed-By: Sam Roberts <[email protected]> Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Jiawen Geng <[email protected]> Reviewed-By: David Carlier <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]> Reviewed-By: Denys Otrishko <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Michael Dawson <[email protected]>

Refs: #30552 (review) PR-URL: #30693 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Jiawen Geng <[email protected]> Reviewed-By: Trivikram Kamat <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

src,doc: add C++ internals documentation

d3d23cd

This aims to help explain some of the internal patterns and utilities that we use. It is by no means exhaustive, and suggestions for additions are welcome. Some of this is based on the existing work from nodejs#26929. Refs: nodejs#26929

addaleax added the c++ Issues and PRs that require attention from people who are familiar with C++. label Nov 20, 2019

nodejs-github-bot added the doc Issues and PRs related to the documentations. label Nov 20, 2019

sam-github approved these changes Nov 20, 2019

View reviewed changes

richardlau approved these changes Nov 20, 2019

View reviewed changes

mscdex reviewed Nov 20, 2019

View reviewed changes

src/README.md Outdated Show resolved Hide resolved

gengjiawen reviewed Nov 20, 2019

View reviewed changes

src/README.md Outdated Show resolved Hide resolved

gengjiawen reviewed Nov 20, 2019

View reviewed changes

src/README.md Outdated Show resolved Hide resolved

gengjiawen approved these changes Nov 20, 2019

View reviewed changes

src/README.md Outdated Show resolved Hide resolved

src/README.md Outdated Show resolved Hide resolved

src/README.md Outdated Show resolved Hide resolved

src/README.md Show resolved Hide resolved

src/README.md Show resolved Hide resolved

src/README.md Outdated Show resolved Hide resolved

gengjiawen reviewed Nov 20, 2019

View reviewed changes

src/README.md Outdated Show resolved Hide resolved

gireeshpunathil reviewed Nov 20, 2019

View reviewed changes

src/README.md Show resolved Hide resolved

src/README.md Show resolved Hide resolved

src/README.md Show resolved Hide resolved

addaleax and others added 2 commits November 20, 2019 11:00

fixup! src,doc: add C++ internals documentation

cbc4152

Co-Authored-By: mscdex <[email protected]>

fixup! src,doc: add C++ internals documentation

09386ef

Co-Authored-By: Jiawen Geng <[email protected]>

devnexen approved these changes Nov 20, 2019

View reviewed changes

src/README.md Show resolved Hide resolved

fixup! src,doc: add C++ internals documentation

4ef7a31

fixup! src,doc: add C++ internals documentation

130ef12

joyeecheung approved these changes Nov 20, 2019

View reviewed changes

addaleax added 2 commits November 20, 2019 13:31

fixup! src,doc: add C++ internals documentation

2445c17

fixup! src,doc: add C++ internals documentation

e1ca24c

addaleax force-pushed the cpp-docs branch from 4c89674 to e1ca24c Compare November 20, 2019 12:34

cjihrig approved these changes Nov 20, 2019

View reviewed changes

src/README.md Outdated Show resolved Hide resolved

src/README.md Outdated Show resolved Hide resolved

src/README.md Show resolved Hide resolved

src/README.md Outdated Show resolved Hide resolved

addaleax and others added 2 commits November 20, 2019 17:11

fixup! src,doc: add C++ internals documentation

309fab4

Co-Authored-By: Colin Ihrig <[email protected]>

fixup! src,doc: add C++ internals documentation

f50f971

Co-Authored-By: Colin Ihrig <[email protected]>

cjihrig reviewed Nov 20, 2019

View reviewed changes

src/README.md Outdated Show resolved Hide resolved

fixup! fixup! src,doc: add C++ internals documentation

9848ba5

lundibundi approved these changes Nov 22, 2019

View reviewed changes

src/README.md Show resolved Hide resolved

src/README.md Outdated Show resolved Hide resolved

jasnell approved these changes Nov 22, 2019

View reviewed changes

addaleax and others added 2 commits November 22, 2019 18:06

fixup! src,doc: add C++ internals documentation

7d32e28

Co-Authored-By: Denys Otrishko <[email protected]>

fixup! src,doc: add C++ internals documentation

b49430d

addaleax added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Nov 22, 2019

mhdawson approved these changes Nov 22, 2019

View reviewed changes

addaleax closed this Nov 22, 2019

addaleax deleted the cpp-docs branch November 22, 2019 20:28

vsemozhetbyt reviewed Nov 24, 2019

View reviewed changes

addaleax added a commit to addaleax/node that referenced this pull request Nov 28, 2019

doc: address nits for src/README.md

09bf04f

Refs: nodejs#30552 (review)

addaleax mentioned this pull request Nov 28, 2019

doc: address nits for src/README.md #30693

Closed

3 tasks

sam-github mentioned this pull request Nov 28, 2019

WIP V8 API usage in Node.js #26929

Closed

4 tasks

BridgeAR mentioned this pull request Dec 3, 2019

v13.3.0 proposal #30774

Merged

BethGriggs mentioned this pull request Dec 9, 2019

v12.13.2 proposal #30865

Closed

BethGriggs mentioned this pull request Dec 23, 2019

v12.14.1 proposal #31069

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

src,doc: add C++ internals documentation #30552

src,doc: add C++ internals documentation #30552

addaleax commented Nov 20, 2019 •

edited

Loading

sam-github left a comment

gengjiawen left a comment

addaleax commented Nov 20, 2019

gengjiawen commented Nov 20, 2019 •

edited

Loading

addaleax commented Nov 20, 2019

addaleax commented Nov 20, 2019

joyeecheung left a comment

joyeecheung Nov 20, 2019

addaleax Nov 20, 2019

joyeecheung Nov 20, 2019 •

edited

Loading

gireeshpunathil Nov 20, 2019

addaleax Nov 20, 2019

joyeecheung Nov 20, 2019 •

edited

Loading

addaleax Nov 20, 2019

joyeecheung Nov 20, 2019 •

edited

Loading

addaleax commented Nov 20, 2019

gireeshpunathil commented Nov 22, 2019

lundibundi left a comment

jasnell left a comment

mhdawson left a comment

addaleax commented Nov 22, 2019

vsemozhetbyt Nov 24, 2019

vsemozhetbyt Nov 24, 2019

vsemozhetbyt Nov 24, 2019

src,doc: add C++ internals documentation #30552

src,doc: add C++ internals documentation #30552

Conversation

addaleax commented Nov 20, 2019 • edited Loading

Checklist

sam-github left a comment

Choose a reason for hiding this comment

gengjiawen left a comment

Choose a reason for hiding this comment

addaleax commented Nov 20, 2019

gengjiawen commented Nov 20, 2019 • edited Loading

addaleax commented Nov 20, 2019

addaleax commented Nov 20, 2019

joyeecheung left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joyeecheung Nov 20, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joyeecheung Nov 20, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joyeecheung Nov 20, 2019 • edited Loading

Choose a reason for hiding this comment

addaleax commented Nov 20, 2019

gireeshpunathil commented Nov 22, 2019

lundibundi left a comment

Choose a reason for hiding this comment

jasnell left a comment

Choose a reason for hiding this comment

mhdawson left a comment

Choose a reason for hiding this comment

addaleax commented Nov 22, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

addaleax commented Nov 20, 2019 •

edited

Loading

gengjiawen commented Nov 20, 2019 •

edited

Loading

joyeecheung Nov 20, 2019 •

edited

Loading

joyeecheung Nov 20, 2019 •

edited

Loading

joyeecheung Nov 20, 2019 •

edited

Loading