From af461dd906dd8d16ee5d901ecc80715c1a471e4d Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Wed, 29 Mar 2023 11:42:03 +0200 Subject: [PATCH 001/110] init: rfc about docs-string format --- rfcs/0000-doc-strings.md | 390 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 390 insertions(+) create mode 100644 rfcs/0000-doc-strings.md diff --git a/rfcs/0000-doc-strings.md b/rfcs/0000-doc-strings.md new file mode 100644 index 000000000..1438c8ae9 --- /dev/null +++ b/rfcs/0000-doc-strings.md @@ -0,0 +1,390 @@ +--- +feature: doc-strings +start-date: 2023-03-27 +author: hsjobeki +co-authors: (find a buddy later to help out with the RFC) +shepherd-team: (names, to be nominated and accepted by RFC steering committee) +shepherd-leader: (name to be appointed by RFC steering committee) +related-issues: (will contain links to implementation PRs) +--- + +# Summary +[summary]: #summary + +Standard for Doc-strings + +# Motivation +[motivation]: #motivation + +This RFC aims at improving quality and consistency of in code documentation. (aka Doc-strings) + +The community offers tools and methods to write in-code documentation for functions and other code related atomic expressions. +This functionality is currently utilized to build a subset of documentation for nix functions. (e.g. nixpkgs.lib documentation via tool: `nixdoc`) + +However the quality and consistency of that __docs-strings__ is itself neither well documented nor standardized. + +This RFC aims at achieving better quality and consistency for docs-strings. Because they carry great documentation potential and allow additional documentation improvements. + +That would allow native nix support, documentation-team or community-driven solutions for automatically generating documentation from them. + +This RFC is intended to be the central place where doc-strings are specified. + +__Current issue__ + +> The existing doc-strings heavily depend on a tool called `nixdoc` and not vice versa. +> +> Instead I want to provide a common standard that every nix user can refer to. + +Everything until now is just a draft if you can provide better ideas e.g. using different formats or syntax please let me know. + +> This RFC aims for general rules for doc-strings. +> Features like: "what different sections exist" and if the might have complex rules (e.g. type: syntax) is not specified. +> +> This RFC aims at providing a formal skeleton, where sections can be extended by the nix community + +# Detailed design +[design]: #detailed-design + +The following abstract rules describe how to write doc-strings. + +The rules are partially derived from the sections below where alternative solutions are considered. So make sure to read them as well. + +> I'am very happy if you comment about whether we should use `## {content}` or `/** {content} */` +> I did write this RFC tailored towards `##` but using `/** */` is still an open discussion. + +We must find one solution out of the following: + +| | 0 `##` | 1 `/** */` | +|---|---|---| +| 0 `# {Keyword}` | `## # Example` | `/** # Example */` | +| 1 `@{Keyword}:` | `# @Exmaple:` |`/** @Example: */` | + +Proposed Solution (0,0) => `##` For docstring body and markdown headings `# H1` + +### Format Rules + +- [F100] - Docs-string are all comments. That start with `##` or `#!` e.g. `## {content}` + +- [F200] - Doc-strings always document / relate to an expression. +- [F201] - Doc-strings starting with `##` relate to the expression in the next line / or more precisely to the next node in the AST. (Details follow, as this might be non-trivial) +- [F202] - Doc-strings that are at the top of a file and that start with `#!` describe the expression exported from the whole file. (Previous node in AST) + +- [F300] - The docstring is continued in the next line if it starts with `##` as well. Leading whitespace are allowed. + +### Structural Rules + +- [S010] - The content of a doc-string is Markdown. + +- [S021] - Content before the first [optional] section is called `description`. + +- [S022] - Headings H1 are reserved markdown headings. Which are specified in [this list](#keywords). Users are allowed to only use H2 (or higher) headings to their free use. +- [S012] - Every [optional] section started by an H1 heading is continued until the next heading starts. To the very end of the comment otherwise. +- [S014] - Every section defines its own rules while they must be compatible with the formal requirements of doc-strings (this RFC) the can override formal rules locally. (e.g. disable markdown, use custom syntax etc.) +- [S017] - Only the H1-sections (`Keywords`) described in in this RFC do exist. (See [the list](#keywords)) +- [S018] - In case of extension, every new section `Keyword` must be added to this RFC first. +- [S030] - If sections follow complex logic it is embraced to specify that logic in an separate sub-RFC. +- [S040] - Usage of the described sections is totally OPTIONAL. +- ... more tbd. + +## Keywords +[keywords]: #keywords + +I wanted to keep the list of initial keywords short. So by the time this RFC focuses on the formal aspects of doc-strings first. More keywords and features for them can be added later on. + +| Keyword | Description | Note | +| --- | --- | --- | +| `Example` | Starts the Example-block. Often contains comprehensive code examples | | +| `Type` | Start the Type-block, it is any free text | Syntax may eventually be specified in the future. [preview](https://typednix.dev). | + +## Decision reasons + +### Why use `##` ? + +This is a quite big change to the existing documentation convention. The reason behind this: Better do it right when always being downwards compatible holds you back. We created a pro-con list in the sections below. + +### Additional `#` + +This provides distinction between regular comments and those who are actually doc-strings. + +### Start with description + +This allows for quick writing without the need for any complex block building + +### Block continuation + +This works like `Markdown` and allows for intuitive usage without knowledge of complex syntax rules. + +### Blocks appear only once + +This reduces complexity and avoids confusion if there is only one place for every concern. + +### All Keywords live in this RFC + +This ensures every keyword is documented. + +### New Keywords + +Ensures newly introduced keyword are discussed in this context first. + +If they bring in more complex sub-features (like Types) those can be discussed in separate RFCs which are back-linked in the [keywords](#keywords) table. This helps to keep this overarching proposal clean and short. + +# Examples and Interactions +[examples-and-interactions]: #examples-and-interactions + +The following example illustrates the structure of doc-strings + +- starting with `##` +- doesn't change the nix syntax + +```nix +# Example - structure + + +## <- Description content-> +## @Example: +## <- Some comprehensive code examples -> +## @Type: +## <- Type -> +``` + +Example: old docs-strings. (To be changed by this RFC) + +```nix +# lib/attrsets.nix + +/* Create a new attribute set with `value` set at the nested attribute location specified in `attrPath`. + Example: + setAttrByPath ["a" "b"] 3 + => { a = { b = 3; }; } + Type: + setAttrByPath :: [String] -> Any -> AttrSet + */ + setAttrByPath +``` + +Example: After changes. + +```nix +# lib/attrsets.nix + +## Create a new attribute set with `value` set at the nested attribute location specified in `attrPath`. +## +## # Example +## setAttrByPath ["a" "b"] 3 +## => { a = { b = 3; }; } +## +## # Type +## setAttrByPath :: [String] -> Any -> AttrSet + setAttrByPath +``` + +## Why change the existing block specifiers? + +First of all: There are no actual block specifiers within nix or nixpkgs. The existing blocks heavily depend on a tool called `nixdoc` and not vice versa. + +-> See [github:nix-community/nixdoc](https://github.com/nix-community/nixdoc) + +> `nixdoc` MUST be changed to support this RFC. (See [Future work](#future-work)) + +The sequence `Example:` has some drawbacks when it comes to syntax: + +1. It is possible that this sequence occurs in a natural text without the intention to start a new docs-string block. +2. It doesn't visually stand out. +3. It is bad that the line needs to start with `Example:` to be valid syntax. Although it is a good practice while writing comments. This shouldn't be syntactically required. > (`nixdoc` requires it). + +## Interactions + +Why doc-strings are valuable + +Doc-strings can be attached to AST nodes without having effect on the actual compile-, evaluation- or build-time because they are just comments. Specialized tools can handle those comments and create static documentation from them. Also integration with LSP is possible. + +# Drawbacks +[drawbacks]: #drawbacks + +- Changes the existing comments inside the code base. + +This could mostly be automated. (e.g via codemod) + +Also this affects only the `lib` folder and few other places that are currently used to build the documentation. + +# Alternatives +[alternatives]: #alternatives + +- By not implementing this feature, nix looses the ability for tool generated documentation. + +- Documentation within code will remain unstable / only determined by nixdoc. + +## Alternative formats + +`##` inspired from rust's `///`. There is the alternative Format using `##` + +Example: + +```nix +# somefile.nix + + ## + ## + ## # Example + ## + ## + ## + ## # Type + ## + ## + mapAttrs = f: s: #... +``` + +| Pro | Con | +|---|---| +| Saves vertical space | Needs Autocompletion (Language Server) to continue the next line. Hustle otherwise to start every line by hand | +| | Changes the existing convention | +| Doesn't need termination (e.g. */) | Can break when interrupted with newlines / non-docstring line beginnings | +| Easier to read / indentation is clear | Multiple comment tokens must be concatenated (Might be more complex) | +| Block continuation is more intuitive (With autocomplete properly setup) | | +| Uses less punctuations and special characters thus is visually more clear and requires less finger spread movements for reaching / and * and @ (for sections) | | +| Works nicely with Markdown content as indentation is visually more clear | Many `#` symbols might be confusing | + +`/** */` In comparison arguments for using `/** */` together with `@{keyword}:` to start sections + +Example: + +```nix + /** + + + @Example: + + + + @Type: + + + */ + mapAttrs = f: s: #... +``` + + +| Pro | Con | +|---|---| +| Clear termination | Takes up more vertical space | +| Doesn't change the existing convention by much | doesn't visually stand out by much (just one more `*` ) | +| Mostly stays compatible with existing implementations | Multiple blocks are not concatenated. They need to be continued | +| No configuration required (LSP for autocompletion on newlines) | | +| | Indentation is not clear / more complex. e.g. The indentation of the first contentfull line is defined as 0, this is how "Noogle" currently works) LSPs could highlight this or show \| (vertical lines) | + +## Candidates not considered + +Javadoc style + +```java + /** + * A short description + * @author Stefan Schneider + * @version 1.1 + * @see https://some.url + */ + public class Product { + ... + } +``` + +Although this has already sneaked into some nix comments. This format is not considered best practice for a variety of good reasons. + +1. Starting every line with `*` creates visual conflicts with markdown bullet list also starting with `*` +2. Takes up more space and needs autocompletion to fill missing `*` beginnings when extended. +3. Pro: Indentation within is clear. +4. Most nix users cannot identify with java or javascript. They like predictable behavior. +5. Essentially combines the cons of both worlds. + +## Alternative section headings + +### Use markdown headings instead + +Example: + +```nix +# somefile.nix + + ## + ## + ## # Example + ## + ## + ## + ## # Type + ## + ## + mapAttrs = f: s: #... +``` + +| Pro | Con | +|---|---| +| Markdown is simple | doesn't visually distinguish from `##` starting the doc-string | +| Using headings feels natural | Users may accidentally use those headings within natural language | +| | Markdown recommends using newlines before and after headings which takes up a lot of vertical space | + +### Use custom headings instead + +Example: + +```nix +# somefile.nix + + ## + ## + ## @Example: + ## + ## + ## + ## @Type: + ## + ## + mapAttrs = f: s: #... +``` + +| Pro | Con | +|---|---| +| Visually stands out | Is new syntax. Where markdown could be more intuitive. doc-strings already are markdown. So why not use markdown | +| Follows more closely the current convention | | +| Needs less vertical space | | +| doesn't need newlines, everything could be even within a single line, allowing compression (may not be needed ?) | | + +# Unresolved questions +[unresolved]: #unresolved-questions + +- Will `nix` itself implement native support like in rust -> `cargo doc` + +- How can a tool keep the connection from where a docstring was defined and where the attribute was exposed (lib/default.nix exposes mapAttrs which is defined at lib/attrsets.nix) + - There are more complicated things. + +-> Answer: A Tool might be able to keep track of a percentage of expressions. Sometimes it may be very hard or impossible. For that case the doc-string can offer a dedicated Keyword that allows to override the scope. + +e.g. + +The following is just an idea for a problem that will arise if tools try to track positions of doc-strings and the location in the nixpkgs tree. (Although this problem is not nixpkgs specific) + +```nix +#untrackable.nix +/*! + This file is called somewhere that cannot be automatically tracked / is impossible to analyse statically. + The 'TreePath' override can be used by the docstring author to set a fixed path in the nixpkgs expression. + (This behavior will not be specified in this RFC) + @TreePath: pkgs.stdenv +*/ +{...}: +{ + # returns something +} +``` + +# Future work +[future]: #future-work + +- When extending nixdoc and/or writing dedicated parsers the following persons can assist: [@hsjobeki] + +- There is an RFC under construction, that specifies the used syntax within the `Type`-Block. It depends on this RFC, as this RFC is the groundwork to provide a standardized field where additional rules can apply. Core-Team: [@hsjobeki] + +- `NixOS/nix` should implement native support for doc-strings. That way our users don't have to rely on nixpkgs or external tools. Those tools can still exist and provide more custom functionality, but it should be natively possible to document your nix expressions. + +- Every existing and future tool can implement against this RFC and rely on it. From dee60aa4073c2d4723544af58783ee04abe0f8eb Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Wed, 29 Mar 2023 12:01:52 +0200 Subject: [PATCH 002/110] init: rfc number --- rfcs/{0000-doc-strings.md => 0145-doc-strings.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename rfcs/{0000-doc-strings.md => 0145-doc-strings.md} (100%) diff --git a/rfcs/0000-doc-strings.md b/rfcs/0145-doc-strings.md similarity index 100% rename from rfcs/0000-doc-strings.md rename to rfcs/0145-doc-strings.md From 450883138f65827e42b063953a5453e4a8c9546d Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Wed, 29 Mar 2023 18:03:36 +0200 Subject: [PATCH 003/110] fix some major spelling issues --- rfcs/0145-doc-strings.md | 149 +++++++++++++++++++++++++-------------- 1 file changed, 97 insertions(+), 52 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 1438c8ae9..5eadbf7e9 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -13,42 +13,83 @@ related-issues: (will contain links to implementation PRs) Standard for Doc-strings +Finally distinguish between doc-strings and comments. + # Motivation [motivation]: #motivation -This RFC aims at improving quality and consistency of in code documentation. (aka Doc-strings) - -The community offers tools and methods to write in-code documentation for functions and other code related atomic expressions. -This functionality is currently utilized to build a subset of documentation for nix functions. (e.g. nixpkgs.lib documentation via tool: `nixdoc`) +This RFC aims at improving consistency of in code documentation. (aka Doc-strings) -However the quality and consistency of that __docs-strings__ is itself neither well documented nor standardized. +The community offers tools and methods (such as nixdoc, nix-doc, etc.) to write and process in-code documentation for functions and other code related atomic expressions. +This functionality is currently utilized to build a subset of documentation for nix functions. (e.g. nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) +There is also [noogle](https://noogle.dev) that indexes subsets of nixpkgs based on the current comments. -This RFC aims at achieving better quality and consistency for docs-strings. Because they carry great documentation potential and allow additional documentation improvements. +However the format of that __docs-strings__ is itself neither well documented nor standardized. -That would allow native nix support, documentation-team or community-driven solutions for automatically generating documentation from them. +This RFC aims at achieving consistency for docs-strings and allow to differentiate between regular comments and doc-strings. -This RFC is intended to be the central place where doc-strings are specified. +We could envision native nix support, documentation-team or community-driven solutions for automatically generating documentation from them. __Current issue__ > The existing doc-strings heavily depend on a tool called `nixdoc` and not vice versa. > -> Instead I want to provide a common standard that every nix user can refer to. +> Instead we want to provide a common standard that every nix user can refer to. Everything until now is just a draft if you can provide better ideas e.g. using different formats or syntax please let me know. +## Example of the current format: + +```nix +/* + + Type: + +*/ +expr = +# Describes 'arg' +arg: +# Describes 'foo' +foo: +``` + > This RFC aims for general rules for doc-strings. -> Features like: "what different sections exist" and if the might have complex rules (e.g. type: syntax) is not specified. +> Features like: "what different sections exist" and if the might have complex rules (e.g. type: syntax) is not specified. > -> This RFC aims at providing a formal skeleton, where sections can be extended by the nix community +> Providing a formal skeleton, where sections can be extended by the nix community + +## Proposed solution + +```nix +# somefile.nix + + ## + ## + ## # Example + ## + ## + ## + ## # Type + ## + ## + mapAttrs = f: s: #... +``` + +This is considered a possible solution because the following reasons: + +- Good visual distinction between doc-strings and comments +- Clear rules and structure +- Saves vertical space +- Doesn't need termination (*/) +- Clear indentation rules. Following the __markdown__ convention. # Detailed design [design]: #detailed-design The following abstract rules describe how to write doc-strings. -The rules are partially derived from the sections below where alternative solutions are considered. So make sure to read them as well. - > I'am very happy if you comment about whether we should use `## {content}` or `/** {content} */` > I did write this RFC tailored towards `##` but using `/** */` is still an open discussion. @@ -57,7 +98,7 @@ We must find one solution out of the following: | | 0 `##` | 1 `/** */` | |---|---|---| | 0 `# {Keyword}` | `## # Example` | `/** # Example */` | -| 1 `@{Keyword}:` | `# @Exmaple:` |`/** @Example: */` | +| 1 `@{Keyword}:` | `# @Example:` |`/** @Example: */` | Proposed Solution (0,0) => `##` For docstring body and markdown headings `# H1` @@ -65,22 +106,40 @@ Proposed Solution (0,0) => `##` For docstring body and markdown headings `# H1` - [F100] - Docs-string are all comments. That start with `##` or `#!` e.g. `## {content}` -- [F200] - Doc-strings always document / relate to an expression. +This is a quite big change to the existing documentation convention. The reason behind this: Better do it right when always being downwards compatible holds you back. We created a pro-con list in the [alternatives](#alternatives) section below. + +This provides distinction between regular comments and those who are actually doc-strings. + +- [F200] - Doc-strings always document / relate to an expression. + - [F201] - Doc-strings starting with `##` relate to the expression in the next line / or more precisely to the next node in the AST. (Details follow, as this might be non-trivial) -- [F202] - Doc-strings that are at the top of a file and that start with `#!` describe the expression exported from the whole file. (Previous node in AST) +- [F202] - Doc-strings that are at the top of a file and that start with `##?` describe the expression exported from the whole file. (Previous node in AST) -- [F300] - The docstring is continued in the next line if it starts with `##` as well. Leading whitespace are allowed. +- [F300] - The docstring is continued in the next line if it starts with `##` as well. Leading whitespace is allowed. ### Structural Rules - [S010] - The content of a doc-string is Markdown. +Allows for intuitive usage without knowledge of complex syntax rules. + - [S021] - Content before the first [optional] section is called `description`. +This allows for quick writing without the need to use sections. + - [S022] - Headings H1 are reserved markdown headings. Which are specified in [this list](#keywords). Users are allowed to only use H2 (or higher) headings to their free use. + +H1 headings start a section to keep it extendable in the future. Users do not use them so we keep track of all used H1 headings. + - [S012] - Every [optional] section started by an H1 heading is continued until the next heading starts. To the very end of the comment otherwise. - [S014] - Every section defines its own rules while they must be compatible with the formal requirements of doc-strings (this RFC) the can override formal rules locally. (e.g. disable markdown, use custom syntax etc.) -- [S017] - Only the H1-sections (`Keywords`) described in in this RFC do exist. (See [the list](#keywords)) +- [S017] - Only the H1-sections (`Keywords`) described in [this list](#keywords) are valid. + +e.g. + +```nix +``` + - [S018] - In case of extension, every new section `Keyword` must be added to this RFC first. - [S030] - If sections follow complex logic it is embraced to specify that logic in an separate sub-RFC. - [S040] - Usage of the described sections is totally OPTIONAL. @@ -89,31 +148,13 @@ Proposed Solution (0,0) => `##` For docstring body and markdown headings `# H1` ## Keywords [keywords]: #keywords -I wanted to keep the list of initial keywords short. So by the time this RFC focuses on the formal aspects of doc-strings first. More keywords and features for them can be added later on. +We wanted to keep the list of initial keywords short. So by the time this RFC focuses on the formal aspects of doc-strings first. More keywords and features for them can be added later on. | Keyword | Description | Note | | --- | --- | --- | | `Example` | Starts the Example-block. Often contains comprehensive code examples | | | `Type` | Start the Type-block, it is any free text | Syntax may eventually be specified in the future. [preview](https://typednix.dev). | -## Decision reasons - -### Why use `##` ? - -This is a quite big change to the existing documentation convention. The reason behind this: Better do it right when always being downwards compatible holds you back. We created a pro-con list in the sections below. - -### Additional `#` - -This provides distinction between regular comments and those who are actually doc-strings. - -### Start with description - -This allows for quick writing without the need for any complex block building - -### Block continuation - -This works like `Markdown` and allows for intuitive usage without knowledge of complex syntax rules. - ### Blocks appear only once This reduces complexity and avoids confusion if there is only one place for every concern. @@ -139,11 +180,10 @@ The following example illustrates the structure of doc-strings ```nix # Example - structure - ## <- Description content-> -## @Example: +## # Example ## <- Some comprehensive code examples -> -## @Type: +## # Type: ## <- Type -> ``` @@ -210,13 +250,13 @@ Also this affects only the `lib` folder and few other places that are currently # Alternatives [alternatives]: #alternatives -- By not implementing this feature, nix looses the ability for tool generated documentation. +- By not implementing this feature, nix loses the ability for tool generated documentation. - Documentation within code will remain unstable / only determined by nixdoc. -## Alternative formats +## Pros-Cons for the different formats -`##` inspired from rust's `///`. There is the alternative Format using `##` +### `##` inspired from rust's `///` Example: @@ -244,6 +284,9 @@ Example: | Block continuation is more intuitive (With autocomplete properly setup) | | | Uses less punctuations and special characters thus is visually more clear and requires less finger spread movements for reaching / and * and @ (for sections) | | | Works nicely with Markdown content as indentation is visually more clear | Many `#` symbols might be confusing | +| | Starting every line with `##` creates visual conflicts with markdown headings `# {Heading}` | + +### `/** */` inspired from the current multiline strings `/** */` In comparison arguments for using `/** */` together with `@{keyword}:` to start sections @@ -264,14 +307,13 @@ Example: mapAttrs = f: s: #... ``` - | Pro | Con | |---|---| | Clear termination | Takes up more vertical space | | Doesn't change the existing convention by much | doesn't visually stand out by much (just one more `*` ) | | Mostly stays compatible with existing implementations | Multiple blocks are not concatenated. They need to be continued | | No configuration required (LSP for autocompletion on newlines) | | -| | Indentation is not clear / more complex. e.g. The indentation of the first contentfull line is defined as 0, this is how "Noogle" currently works) LSPs could highlight this or show \| (vertical lines) | +| | Indentation is not clear / more complex. | ## Candidates not considered @@ -291,15 +333,15 @@ Javadoc style Although this has already sneaked into some nix comments. This format is not considered best practice for a variety of good reasons. -1. Starting every line with `*` creates visual conflicts with markdown bullet list also starting with `*` +1. Essentially combines the cons of both worlds. 2. Takes up more space and needs autocompletion to fill missing `*` beginnings when extended. -3. Pro: Indentation within is clear. -4. Most nix users cannot identify with java or javascript. They like predictable behavior. -5. Essentially combines the cons of both worlds. +3. Starting every line with `*` creates visual conflicts with markdown bullet list also starting with `*`. +4. Pro: Indentation within is clear. +5. Most nix users cannot identify with java or javascript. They like predictable behavior. -## Alternative section headings +## Pros-Cons for section headings -### Use markdown headings instead +### Markdown headings Example: @@ -323,8 +365,9 @@ Example: | Markdown is simple | doesn't visually distinguish from `##` starting the doc-string | | Using headings feels natural | Users may accidentally use those headings within natural language | | | Markdown recommends using newlines before and after headings which takes up a lot of vertical space | +| | Markdown headings do not visually stand out from lines that already started with `##` | -### Use custom headings instead +### Custom headings `@{Keyword}:` Example: @@ -353,6 +396,8 @@ Example: # Unresolved questions [unresolved]: #unresolved-questions +- `nixodc` offers doc-strings to describe function arguments. This is currently not compatible until some section for `args` are defined. + - Will `nix` itself implement native support like in rust -> `cargo doc` - How can a tool keep the connection from where a docstring was defined and where the attribute was exposed (lib/default.nix exposes mapAttrs which is defined at lib/attrsets.nix) From 34bf50e860e8019d220b3acc80ecfb4fc41a3acb Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Wed, 29 Mar 2023 18:35:57 +0200 Subject: [PATCH 004/110] gramatical cleanup --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 5eadbf7e9..430ca08ad 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -56,7 +56,7 @@ foo: ``` > This RFC aims for general rules for doc-strings. -> Features like: "what different sections exist" and if the might have complex rules (e.g. type: syntax) is not specified. +> Features like: "what different sections exist" and if they might have complex rules (e.g. type: syntax) is not specified. > > Providing a formal skeleton, where sections can be extended by the nix community From 130d9e5af06791732b1b0b71a2cad05b734fb760 Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Wed, 29 Mar 2023 18:37:27 +0200 Subject: [PATCH 005/110] gramatical cleanup --- rfcs/0145-doc-strings.md | 90 +++++++++++++++++----------------------- 1 file changed, 38 insertions(+), 52 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 430ca08ad..75b737edd 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -13,30 +13,30 @@ related-issues: (will contain links to implementation PRs) Standard for Doc-strings -Finally distinguish between doc-strings and comments. +Finally, distinguish between doc-strings and comments. # Motivation [motivation]: #motivation -This RFC aims at improving consistency of in code documentation. (aka Doc-strings) +This RFC aims to improve the consistency of in-code documentation. (aka Doc-strings) -The community offers tools and methods (such as nixdoc, nix-doc, etc.) to write and process in-code documentation for functions and other code related atomic expressions. -This functionality is currently utilized to build a subset of documentation for nix functions. (e.g. nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) -There is also [noogle](https://noogle.dev) that indexes subsets of nixpkgs based on the current comments. +The community offers tools and methods (such as nixdoc, nix-doc, etc.) to write and process in-code documentation for functions and other code-related atomic expressions. +We are currently utilizing this functionality to build a subset of documentation for nix functions. (e.g., nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) +Also, [noogle](https://noogle.dev) indexes subsets of nixpkgs based on multiline comments. -However the format of that __docs-strings__ is itself neither well documented nor standardized. +However, the format of that __docs-strings__ needs to be better documented and standardized. -This RFC aims at achieving consistency for docs-strings and allow to differentiate between regular comments and doc-strings. +This RFC aims to achieve consistency for docs-strings and allows for differentiation between regular comments and doc-strings. -We could envision native nix support, documentation-team or community-driven solutions for automatically generating documentation from them. +We could envision native nix support, a documentation team, or community-driven solutions for automatically generating documentation from them. __Current issue__ > The existing doc-strings heavily depend on a tool called `nixdoc` and not vice versa. > -> Instead we want to provide a common standard that every nix user can refer to. +> Instead, we want to provide a common standard that every nix user can refer to. -Everything until now is just a draft if you can provide better ideas e.g. using different formats or syntax please let me know. +Everything until now is just a draft; if you can provide better ideas e.g. using different formats or syntax please let me know. ## Example of the current format: @@ -56,9 +56,9 @@ foo: ``` > This RFC aims for general rules for doc-strings. -> Features like: "what different sections exist" and if they might have complex rules (e.g. type: syntax) is not specified. +> Features like: "what different sections exist" and if they might have complex rules (e.g., type: syntax) are not specified. > -> Providing a formal skeleton, where sections can be extended by the nix community +> Providing a formal skeleton where the nix community can extend sections ## Proposed solution @@ -77,7 +77,7 @@ foo: mapAttrs = f: s: #... ``` -This is considered a possible solution because the following reasons: +We considered this possible solution because the following reasons: - Good visual distinction between doc-strings and comments - Clear rules and structure @@ -90,7 +90,7 @@ This is considered a possible solution because the following reasons: The following abstract rules describe how to write doc-strings. -> I'am very happy if you comment about whether we should use `## {content}` or `/** {content} */` +> I would be pleased if you comment about whether we should use `## {content}` or `/** {content} */` > I did write this RFC tailored towards `##` but using `/** */` is still an open discussion. We must find one solution out of the following: @@ -106,33 +106,33 @@ Proposed Solution (0,0) => `##` For docstring body and markdown headings `# H1` - [F100] - Docs-string are all comments. That start with `##` or `#!` e.g. `## {content}` -This is a quite big change to the existing documentation convention. The reason behind this: Better do it right when always being downwards compatible holds you back. We created a pro-con list in the [alternatives](#alternatives) section below. +This RFC is a significant change to the existing documentation convention. This is because it is better to do it right when always being downward compatible holds you back. We created a pro-con list in the [alternatives](#alternatives) section below. -This provides distinction between regular comments and those who are actually doc-strings. +We finally need to distinguish between regular comments and doc strings. We found this format to be the most distinguishable. - [F200] - Doc-strings always document / relate to an expression. -- [F201] - Doc-strings starting with `##` relate to the expression in the next line / or more precisely to the next node in the AST. (Details follow, as this might be non-trivial) +- [F201] - Doc-strings starting with `##` relate to the expression in the following line / or, more precisely, to the next node in the AST. (Details follow, as this might be non-trivial) - [F202] - Doc-strings that are at the top of a file and that start with `##?` describe the expression exported from the whole file. (Previous node in AST) -- [F300] - The docstring is continued in the next line if it starts with `##` as well. Leading whitespace is allowed. +- [F300] - The docstring is continued in the following line if it also starts with `##`. Leading whitespace is allowed. ### Structural Rules - [S010] - The content of a doc-string is Markdown. -Allows for intuitive usage without knowledge of complex syntax rules. +It allows for intuitive usage without knowledge of complex syntax rules. - [S021] - Content before the first [optional] section is called `description`. This allows for quick writing without the need to use sections. -- [S022] - Headings H1 are reserved markdown headings. Which are specified in [this list](#keywords). Users are allowed to only use H2 (or higher) headings to their free use. +- [S022] - Headings H1 are reserved markdown headings. Which are specified in [this list](#keywords). Users are allowed to only use H2 (or higher) headings for their free use. -H1 headings start a section to keep it extendable in the future. Users do not use them so we keep track of all used H1 headings. +H1 headings start a section to keep it extendable in the future. Users do not use them, so we keep track of all allowed H1 headings. -- [S012] - Every [optional] section started by an H1 heading is continued until the next heading starts. To the very end of the comment otherwise. -- [S014] - Every section defines its own rules while they must be compatible with the formal requirements of doc-strings (this RFC) the can override formal rules locally. (e.g. disable markdown, use custom syntax etc.) +- [S012] - Every [optional] section started by an H1 heading is continued until the next heading starts. To the very end of the comment, otherwise. +- [S014] - Every section defines its own rules. They must be compatible with the formal requirements of doc-strings (this RFC) that can override formal rules locally. (e.g., disable Markdown, use custom syntax, etc.) - [S017] - Only the H1-sections (`Keywords`) described in [this list](#keywords) are valid. e.g. @@ -140,9 +140,9 @@ e.g. ```nix ``` -- [S018] - In case of extension, every new section `Keyword` must be added to this RFC first. -- [S030] - If sections follow complex logic it is embraced to specify that logic in an separate sub-RFC. -- [S040] - Usage of the described sections is totally OPTIONAL. +- [S018] - In case of extension, every new section `Keyword` must first be added to this RFC. +- [S030] - If sections follow complex logic, it is embraced to specify that logic in a separate sub-RFC. +- [S040] - Usage of the described sections is OPTIONAL. - ... more tbd. ## Keywords @@ -153,21 +153,7 @@ We wanted to keep the list of initial keywords short. So by the time this RFC fo | Keyword | Description | Note | | --- | --- | --- | | `Example` | Starts the Example-block. Often contains comprehensive code examples | | -| `Type` | Start the Type-block, it is any free text | Syntax may eventually be specified in the future. [preview](https://typednix.dev). | - -### Blocks appear only once - -This reduces complexity and avoids confusion if there is only one place for every concern. - -### All Keywords live in this RFC - -This ensures every keyword is documented. - -### New Keywords - -Ensures newly introduced keyword are discussed in this context first. - -If they bring in more complex sub-features (like Types) those can be discussed in separate RFCs which are back-linked in the [keywords](#keywords) table. This helps to keep this overarching proposal clean and short. +| `Type` | Start the Type-block; it is any free text | Syntax may eventually be specified in the future. [preview](https://typednix.dev). | # Examples and Interactions [examples-and-interactions]: #examples-and-interactions @@ -236,7 +222,7 @@ The sequence `Example:` has some drawbacks when it comes to syntax: Why doc-strings are valuable -Doc-strings can be attached to AST nodes without having effect on the actual compile-, evaluation- or build-time because they are just comments. Specialized tools can handle those comments and create static documentation from them. Also integration with LSP is possible. +Doc-strings can be attached to AST nodes without affecting the actual compile-, evaluation- or build-time because they are just comments. Specialized tools can handle those comments and create static documentation from them. Also, integration with LSP is possible. # Drawbacks [drawbacks]: #drawbacks @@ -245,12 +231,12 @@ Doc-strings can be attached to AST nodes without having effect on the actual com This could mostly be automated. (e.g via codemod) -Also this affects only the `lib` folder and few other places that are currently used to build the documentation. +Also, this affects only the `lib` folder and a few other places that are currently used to build the documentation. # Alternatives [alternatives]: #alternatives -- By not implementing this feature, nix loses the ability for tool generated documentation. +- By not implementing this feature, nix loses the ability for tool-generated documentation. - Documentation within code will remain unstable / only determined by nixdoc. @@ -282,11 +268,11 @@ Example: | Doesn't need termination (e.g. */) | Can break when interrupted with newlines / non-docstring line beginnings | | Easier to read / indentation is clear | Multiple comment tokens must be concatenated (Might be more complex) | | Block continuation is more intuitive (With autocomplete properly setup) | | -| Uses less punctuations and special characters thus is visually more clear and requires less finger spread movements for reaching / and * and @ (for sections) | | +| Uses fewer punctuations and special characters thus is visually more clear and requires less finger spread movements for reaching / and * and @ (for sections) | | | Works nicely with Markdown content as indentation is visually more clear | Many `#` symbols might be confusing | | | Starting every line with `##` creates visual conflicts with markdown headings `# {Heading}` | -### `/** */` inspired from the current multiline strings +### `/** */` inspired by the current multiline strings `/** */` In comparison arguments for using `/** */` together with `@{keyword}:` to start sections @@ -335,7 +321,7 @@ Although this has already sneaked into some nix comments. This format is not con 1. Essentially combines the cons of both worlds. 2. Takes up more space and needs autocompletion to fill missing `*` beginnings when extended. -3. Starting every line with `*` creates visual conflicts with markdown bullet list also starting with `*`. +3. Starting every line with `*` creates visual conflicts with the markdown bullet list also starting with `*`. 4. Pro: Indentation within is clear. 5. Most nix users cannot identify with java or javascript. They like predictable behavior. @@ -388,7 +374,7 @@ Example: | Pro | Con | |---|---| -| Visually stands out | Is new syntax. Where markdown could be more intuitive. doc-strings already are markdown. So why not use markdown | +| Visually stands out | Is new syntax. Where Markdown could be more intuitive. doc-strings already are Markdown. So why not use markdown | | Follows more closely the current convention | | | Needs less vertical space | | | doesn't need newlines, everything could be even within a single line, allowing compression (may not be needed ?) | | @@ -396,23 +382,23 @@ Example: # Unresolved questions [unresolved]: #unresolved-questions -- `nixodc` offers doc-strings to describe function arguments. This is currently not compatible until some section for `args` are defined. +- `nixodc` offers doc-strings to describe function arguments. This is currently not compatible until some sections for `args` are defined. - Will `nix` itself implement native support like in rust -> `cargo doc` - How can a tool keep the connection from where a docstring was defined and where the attribute was exposed (lib/default.nix exposes mapAttrs which is defined at lib/attrsets.nix) - There are more complicated things. --> Answer: A Tool might be able to keep track of a percentage of expressions. Sometimes it may be very hard or impossible. For that case the doc-string can offer a dedicated Keyword that allows to override the scope. +-> Answer: A Tool might be able to keep track of a percentage of expressions. Sometimes it may be very hard or impossible. For that case, the doc-string can offer a dedicated Keyword that allows overriding the scope. e.g. -The following is just an idea for a problem that will arise if tools try to track positions of doc-strings and the location in the nixpkgs tree. (Although this problem is not nixpkgs specific) +The following is just an idea for a problem that will arise if tools try to track the positions of doc-strings and the location in the nixpkgs tree. (Although this problem is not nixpkgs specific) ```nix #untrackable.nix /*! - This file is called somewhere that cannot be automatically tracked / is impossible to analyse statically. + This file is called somewhere that cannot be automatically tracked/is impossible to analyze statically. The 'TreePath' override can be used by the docstring author to set a fixed path in the nixpkgs expression. (This behavior will not be specified in this RFC) @TreePath: pkgs.stdenv From ab0aa26744ff069611656f6314d667e8d0d0cf1d Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Wed, 29 Mar 2023 18:38:45 +0200 Subject: [PATCH 006/110] fix: consistent naming: doc-string(s) --- rfcs/0145-doc-strings.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 75b737edd..b6d6f3205 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -24,9 +24,9 @@ The community offers tools and methods (such as nixdoc, nix-doc, etc.) to write We are currently utilizing this functionality to build a subset of documentation for nix functions. (e.g., nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) Also, [noogle](https://noogle.dev) indexes subsets of nixpkgs based on multiline comments. -However, the format of that __docs-strings__ needs to be better documented and standardized. +However, the format of that __doc-strings__ needs to be better documented and standardized. -This RFC aims to achieve consistency for docs-strings and allows for differentiation between regular comments and doc-strings. +This RFC aims to achieve consistency for doc-strings and allows for differentiation between regular comments and doc-strings. We could envision native nix support, a documentation team, or community-driven solutions for automatically generating documentation from them. @@ -104,7 +104,7 @@ Proposed Solution (0,0) => `##` For docstring body and markdown headings `# H1` ### Format Rules -- [F100] - Docs-string are all comments. That start with `##` or `#!` e.g. `## {content}` +- [F100] - doc-string are all comments. That start with `##` or `#!` e.g. `## {content}` This RFC is a significant change to the existing documentation convention. This is because it is better to do it right when always being downward compatible holds you back. We created a pro-con list in the [alternatives](#alternatives) section below. @@ -173,7 +173,7 @@ The following example illustrates the structure of doc-strings ## <- Type -> ``` -Example: old docs-strings. (To be changed by this RFC) +Example: old doc-strings. (To be changed by this RFC) ```nix # lib/attrsets.nix @@ -214,7 +214,7 @@ First of all: There are no actual block specifiers within nix or nixpkgs. The ex The sequence `Example:` has some drawbacks when it comes to syntax: -1. It is possible that this sequence occurs in a natural text without the intention to start a new docs-string block. +1. It is possible that this sequence occurs in a natural text without the intention to start a new doc-string block. 2. It doesn't visually stand out. 3. It is bad that the line needs to start with `Example:` to be valid syntax. Although it is a good practice while writing comments. This shouldn't be syntactically required. > (`nixdoc` requires it). From 2b0dea2696a60697075df1606388f6e086acb4b2 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Wed, 29 Mar 2023 18:51:38 +0200 Subject: [PATCH 007/110] Update rfcs/0145-doc-strings.md Co-authored-by: figsoda --- rfcs/0145-doc-strings.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index b6d6f3205..eaa868e27 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -65,16 +65,18 @@ foo: ```nix # somefile.nix - ## - ## - ## # Example - ## - ## - ## - ## # Type - ## - ## - mapAttrs = f: s: #... +{ + ## + ## + ## # Example + ## + ## + ## + ## # Type + ## + ## + mapAttrs = f: s: <...>; +} ``` We considered this possible solution because the following reasons: From a3ee89a13612ff44ae05faae74c477fbeff6b7b1 Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Wed, 29 Mar 2023 19:02:32 +0200 Subject: [PATCH 008/110] make statement more clear --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index eaa868e27..3031cf8ca 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -131,7 +131,7 @@ This allows for quick writing without the need to use sections. - [S022] - Headings H1 are reserved markdown headings. Which are specified in [this list](#keywords). Users are allowed to only use H2 (or higher) headings for their free use. -H1 headings start a section to keep it extendable in the future. Users do not use them, so we keep track of all allowed H1 headings. +H1 headings start a section to keep it extendable in the future. Users do not choose them freely, so we keep track of all allowed H1 headings. - [S012] - Every [optional] section started by an H1 heading is continued until the next heading starts. To the very end of the comment, otherwise. - [S014] - Every section defines its own rules. They must be compatible with the formal requirements of doc-strings (this RFC) that can override formal rules locally. (e.g., disable Markdown, use custom syntax, etc.) From e1a747585f52389e0f78ab72e9b47d4bc706a44e Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Wed, 29 Mar 2023 19:04:59 +0200 Subject: [PATCH 009/110] add statement what a sections is --- rfcs/0145-doc-strings.md | 1 + 1 file changed, 1 insertion(+) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 3031cf8ca..0b556c432 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -125,6 +125,7 @@ We finally need to distinguish between regular comments and doc strings. We foun It allows for intuitive usage without knowledge of complex syntax rules. +- [S011] - predefined heading `keywords` may start a section. - [S021] - Content before the first [optional] section is called `description`. This allows for quick writing without the need to use sections. From d7e0cf517ff1d2608d617a9c4c8d5f40ce444185 Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Wed, 29 Mar 2023 19:05:32 +0200 Subject: [PATCH 010/110] remove empty example --- rfcs/0145-doc-strings.md | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 0b556c432..eae4746f4 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -137,12 +137,7 @@ H1 headings start a section to keep it extendable in the future. Users do not ch - [S012] - Every [optional] section started by an H1 heading is continued until the next heading starts. To the very end of the comment, otherwise. - [S014] - Every section defines its own rules. They must be compatible with the formal requirements of doc-strings (this RFC) that can override formal rules locally. (e.g., disable Markdown, use custom syntax, etc.) - [S017] - Only the H1-sections (`Keywords`) described in [this list](#keywords) are valid. - -e.g. - -```nix -``` - + - [S018] - In case of extension, every new section `Keyword` must first be added to this RFC. - [S030] - If sections follow complex logic, it is embraced to specify that logic in a separate sub-RFC. - [S040] - Usage of the described sections is OPTIONAL. From 4d69893b1fc8296d241c0f275059b8d78c46850f Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 30 Mar 2023 09:29:00 +0200 Subject: [PATCH 011/110] Update rfcs/0145-doc-strings.md Co-authored-by: figsoda --- rfcs/0145-doc-strings.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index eae4746f4..7aaf2eb0c 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -164,11 +164,11 @@ The following example illustrates the structure of doc-strings ```nix # Example - structure -## <- Description content-> -## # Example -## <- Some comprehensive code examples -> -## # Type: -## <- Type -> +## <- Description content-> +## # Example +## <- Some comprehensive code examples -> +## # Type: +## <- Type -> ``` Example: old doc-strings. (To be changed by this RFC) From b4a12973e91c76e5ccf0c38736fbefc13c447641 Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Thu, 30 Mar 2023 09:45:32 +0200 Subject: [PATCH 012/110] add: more examples, explanations --- rfcs/0145-doc-strings.md | 99 ++++++++++++++++++---------------------- 1 file changed, 45 insertions(+), 54 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 7aaf2eb0c..db085ddb1 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -38,7 +38,7 @@ __Current issue__ Everything until now is just a draft; if you can provide better ideas e.g. using different formats or syntax please let me know. -## Example of the current format: +## Example of the current format ```nix /* @@ -115,9 +115,44 @@ We finally need to distinguish between regular comments and doc strings. We foun - [F200] - Doc-strings always document / relate to an expression. - [F201] - Doc-strings starting with `##` relate to the expression in the following line / or, more precisely, to the next node in the AST. (Details follow, as this might be non-trivial) -- [F202] - Doc-strings that are at the top of a file and that start with `##?` describe the expression exported from the whole file. (Previous node in AST) +- [F202] - Doc-strings that are at the top of a file and that start with `#|` describe the expression exported from the whole file. (Previous node in AST) -- [F300] - The docstring is continued in the following line if it also starts with `##`. Leading whitespace is allowed. +In comparison; rustdoc uses `//!`. But using `#!` is considered a bead idea, as it can be confused with identical bash shebangs `#!`. + +> This is not final yet. If you have any ideas let us know in the comments. + +Example of a comment referring to the whole file: + +```nix +#| +#| +#| # Example +#| +#| # Type +#| The Type of the expression returned by the file +{ + id = x: x +} +``` + +- [F300] - The docstring is continued in the following line if it also starts with `##` / `#|`. Leading whitespace is allowed. + +Example: docstring continuation + +```nix +## Doc-string A +## .... +## This block has no expression in the next line. +## Therefore it doesn't have any effect +## .... +## Doc-string A + + +## Doc-string B +## -- this block documents the purpose of '1' +## Doc-string B +1 +``` ### Structural Rules @@ -153,56 +188,7 @@ We wanted to keep the list of initial keywords short. So by the time this RFC fo | `Example` | Starts the Example-block. Often contains comprehensive code examples | | | `Type` | Start the Type-block; it is any free text | Syntax may eventually be specified in the future. [preview](https://typednix.dev). | -# Examples and Interactions -[examples-and-interactions]: #examples-and-interactions - -The following example illustrates the structure of doc-strings - -- starting with `##` -- doesn't change the nix syntax - -```nix -# Example - structure - -## <- Description content-> -## # Example -## <- Some comprehensive code examples -> -## # Type: -## <- Type -> -``` - -Example: old doc-strings. (To be changed by this RFC) - -```nix -# lib/attrsets.nix - -/* Create a new attribute set with `value` set at the nested attribute location specified in `attrPath`. - Example: - setAttrByPath ["a" "b"] 3 - => { a = { b = 3; }; } - Type: - setAttrByPath :: [String] -> Any -> AttrSet - */ - setAttrByPath -``` - -Example: After changes. - -```nix -# lib/attrsets.nix - -## Create a new attribute set with `value` set at the nested attribute location specified in `attrPath`. -## -## # Example -## setAttrByPath ["a" "b"] 3 -## => { a = { b = 3; }; } -## -## # Type -## setAttrByPath :: [String] -> Any -> AttrSet - setAttrByPath -``` - -## Why change the existing block specifiers? +## Why change the existing section specifiers? First of all: There are no actual block specifiers within nix or nixpkgs. The existing blocks heavily depend on a tool called `nixdoc` and not vice versa. @@ -212,7 +198,7 @@ First of all: There are no actual block specifiers within nix or nixpkgs. The ex The sequence `Example:` has some drawbacks when it comes to syntax: -1. It is possible that this sequence occurs in a natural text without the intention to start a new doc-string block. +1. It is possible that this sequence occurs in a natural text without the intention to start a new doc-string section. 2. It doesn't visually stand out. 3. It is bad that the line needs to start with `Example:` to be valid syntax. Although it is a good practice while writing comments. This shouldn't be syntactically required. > (`nixdoc` requires it). @@ -222,6 +208,11 @@ Why doc-strings are valuable Doc-strings can be attached to AST nodes without affecting the actual compile-, evaluation- or build-time because they are just comments. Specialized tools can handle those comments and create static documentation from them. Also, integration with LSP is possible. +Many files within nixpkgs contain detailed comments we cannot currently use. +An example: [make-disk-image.nix](https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-disk-image.nix) + +Following this RFC means refactoring for existing comments, but it also means that we can finally use all comments that were intended to be doc-strings + # Drawbacks [drawbacks]: #drawbacks From f6bd84fdc11ad75f60188094e47e2f248290f7c5 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 30 Mar 2023 14:50:20 +0200 Subject: [PATCH 013/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 80 ++++++++++++++++++++++++++++------------ 1 file changed, 57 insertions(+), 23 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index db085ddb1..15dd980f7 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -11,9 +11,11 @@ related-issues: (will contain links to implementation PRs) # Summary [summary]: #summary -Standard for Doc-strings +Standard for Docstring format -Finally, distinguish between doc-strings and comments. +Wikipedia states: + +> In programming, a docstring is a string literal specified in source code that is used, like a comment, to document a specific segment of code. Unlike conventional source code comments, or even specifically formatted comments like docblocks, docstrings are not stripped from the source tree when it is parsed and are retained throughout the runtime of the program. This allows the programmer to inspect these comments at run time, for instance as an interactive help system, or as metadata. # Motivation [motivation]: #motivation @@ -30,13 +32,20 @@ This RFC aims to achieve consistency for doc-strings and allows for differentiat We could envision native nix support, a documentation team, or community-driven solutions for automatically generating documentation from them. +More specifically, we envision the following possible features: (Long term) + +- Automatically build documentation - At most atomic things that directly relate to one specific piece of code (e.g., mkDerivation, make-disk-image, lib-functions, etc.) +- Hover information in IDEs +- Autocomplete in IDEs +- Detect broken code/misuse of specific functionality. + __Current issue__ -> The existing doc-strings heavily depend on a tool called `nixdoc` and not vice versa. +> The existing doc-strings heavily depend on a tool called `nixdoc` not vice versa. > > Instead, we want to provide a common standard that every nix user can refer to. -Everything until now is just a draft; if you can provide better ideas e.g. using different formats or syntax please let me know. +Everything until now is just a draft; if you can provide better ideas, e.g. using different formats or syntax, please let me know. ## Example of the current format @@ -62,6 +71,10 @@ foo: ## Proposed solution +In general we propose two kinds of doc-strings + +## Doc-string referencing the subsequent expression + ```nix # somefile.nix @@ -79,6 +92,25 @@ foo: } ``` +## Doc-string referencing the file expression + +```nix +#| +#| +#| # Example +#| +#| +#| +#| # Type +#| +#| + +{...}: +{ + mapAttrs = f: s: <...>; +} +``` + We considered this possible solution because the following reasons: - Good visual distinction between doc-strings and comments @@ -117,9 +149,9 @@ We finally need to distinguish between regular comments and doc strings. We foun - [F201] - Doc-strings starting with `##` relate to the expression in the following line / or, more precisely, to the next node in the AST. (Details follow, as this might be non-trivial) - [F202] - Doc-strings that are at the top of a file and that start with `#|` describe the expression exported from the whole file. (Previous node in AST) -In comparison; rustdoc uses `//!`. But using `#!` is considered a bead idea, as it can be confused with identical bash shebangs `#!`. +In comparison, rustdoc uses `//!`. But using `#!` is considered a bad idea, as it can be confused with identical bash shebangs `#!`. -> This is not final yet. If you have any ideas let us know in the comments. +> This is still being determined. If you have any ideas, let us know in the comments. Example of a comment referring to the whole file: @@ -255,10 +287,10 @@ Example: | Saves vertical space | Needs Autocompletion (Language Server) to continue the next line. Hustle otherwise to start every line by hand | | | Changes the existing convention | | Doesn't need termination (e.g. */) | Can break when interrupted with newlines / non-docstring line beginnings | -| Easier to read / indentation is clear | Multiple comment tokens must be concatenated (Might be more complex) | -| Block continuation is more intuitive (With autocomplete properly setup) | | -| Uses fewer punctuations and special characters thus is visually more clear and requires less finger spread movements for reaching / and * and @ (for sections) | | -| Works nicely with Markdown content as indentation is visually more clear | Many `#` symbols might be confusing | +| Easier to read / Indentation is clear | Multiple comment tokens must be concatenated (Might be more complex) | +| Block continuation is more intuitive (With autocomplete setup properly) | | +| Uses fewer punctuations and special characters; thus is visually more clear and requires less finger spread movements for reaching / and * and @ (for sections) | | +| Works nicely with Markdown content as Indentation is visually more clear | Many `#` symbols might be confusing | | | Starting every line with `##` creates visual conflicts with markdown headings `# {Heading}` | ### `/** */` inspired by the current multiline strings @@ -288,7 +320,7 @@ Example: | Doesn't change the existing convention by much | doesn't visually stand out by much (just one more `*` ) | | Mostly stays compatible with existing implementations | Multiple blocks are not concatenated. They need to be continued | | No configuration required (LSP for autocompletion on newlines) | | -| | Indentation is not clear / more complex. | +| | Indentation needs to be clarified / more complex. | ## Candidates not considered @@ -309,7 +341,7 @@ Javadoc style Although this has already sneaked into some nix comments. This format is not considered best practice for a variety of good reasons. 1. Essentially combines the cons of both worlds. -2. Takes up more space and needs autocompletion to fill missing `*` beginnings when extended. +2. It Takes up more space and needs autocompletion to fill missing `*` beginnings when extended. 3. Starting every line with `*` creates visual conflicts with the markdown bullet list also starting with `*`. 4. Pro: Indentation within is clear. 5. Most nix users cannot identify with java or javascript. They like predictable behavior. @@ -363,35 +395,37 @@ Example: | Pro | Con | |---|---| -| Visually stands out | Is new syntax. Where Markdown could be more intuitive. doc-strings already are Markdown. So why not use markdown | +| Visually stands out | Is new syntax. Where Markdown could be more intuitive. Doc-strings already are Markdown. So why not use markdown | | Follows more closely the current convention | | | Needs less vertical space | | | doesn't need newlines, everything could be even within a single line, allowing compression (may not be needed ?) | | +## Alternative approach - just comments + +There is the idea from python that doc-strings are just strings, not even special ones. Comments will be docstrings if they follow specific placement rules. However, we thought this was a bad idea to follow. Such complex placement rules require the users to understand where those places are; with nix syntax, this is slightly more complex than with python. Because we don't have keywords such as `class MyClass():` or `def function():` where placement would be obvious + # Unresolved questions [unresolved]: #unresolved-questions -- `nixodc` offers doc-strings to describe function arguments. This is currently not compatible until some sections for `args` are defined. +- `nixodc` offers comments to describe function arguments. This is currently not compatible until some sections for `args` are defined. - Will `nix` itself implement native support like in rust -> `cargo doc` - How can a tool keep the connection from where a docstring was defined and where the attribute was exposed (lib/default.nix exposes mapAttrs which is defined at lib/attrsets.nix) - There are more complicated things. --> Answer: A Tool might be able to keep track of a percentage of expressions. Sometimes it may be very hard or impossible. For that case, the doc-string can offer a dedicated Keyword that allows overriding the scope. +-> Answer: A Tool might be able to keep track of a percentage of expressions, and sometimes it may be very hard or impossible. For that case, the doc-string can offer a dedicated Keyword to override the scope. e.g. -The following is just an idea for a problem that will arise if tools try to track the positions of doc-strings and the location in the nixpkgs tree. (Although this problem is not nixpkgs specific) +The following is an idea for a problem that will arise if tools try to track doc-strings' positions and the location in the nixpkgs tree. (Although this problem is not nixpkgs specific) ```nix -#untrackable.nix -/*! - This file is called somewhere that cannot be automatically tracked/is impossible to analyze statically. - The 'TreePath' override can be used by the docstring author to set a fixed path in the nixpkgs expression. - (This behavior will not be specified in this RFC) - @TreePath: pkgs.stdenv -*/ +#| This file is called somewhere that cannot be automatically tracked/is impossible to analyze statically. +#| The 'TreePath' override can be used by the docstring author to set a fixed path in the nixpkgs expression. +#| (This behavior will not be specified in this RFC) +#| @TreePath: pkgs.stdenv + {...}: { # returns something From 4d2329ab4dbda70e41dc3efd21653c1a5a8fd229 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 30 Mar 2023 15:05:45 +0200 Subject: [PATCH 014/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 15dd980f7..556d34aff 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -24,9 +24,16 @@ This RFC aims to improve the consistency of in-code documentation. (aka Doc-stri The community offers tools and methods (such as nixdoc, nix-doc, etc.) to write and process in-code documentation for functions and other code-related atomic expressions. We are currently utilizing this functionality to build a subset of documentation for nix functions. (e.g., nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) -Also, [noogle](https://noogle.dev) indexes subsets of nixpkgs based on multiline comments. -However, the format of that __doc-strings__ needs to be better documented and standardized. +Other tools that work directly with the nix AST and comments: + +- [noogle](https://noogle.dev) - Nix API search engine. It allows you to search functions and other expressions. +- [nix-doc](https://github.com/lf-/nix-doc) - A Nix developer tool leveraging the rnix Nix parser for intelligent documentation search and tags generation +- [manix](https://github.com/mlvzk/manix) - A fast CLI documentation searcher for Nix. + +All those tools would profit from the proposed changes. + +So the format of that __doc-strings__ needs to be better documented and standardized. This RFC aims to achieve consistency for doc-strings and allows for differentiation between regular comments and doc-strings. @@ -67,7 +74,7 @@ foo: > This RFC aims for general rules for doc-strings. > Features like: "what different sections exist" and if they might have complex rules (e.g., type: syntax) are not specified. > -> Providing a formal skeleton where the nix community can extend sections +> Providing a formal skeleton where the nix community can extend sections. ## Proposed solution @@ -75,11 +82,14 @@ In general we propose two kinds of doc-strings ## Doc-string referencing the subsequent expression +Uses `##` to create visual blocks and draw attention to documentation. + ```nix # somefile.nix { ## + ## Documentation for 'mapAttrs' ## ## # Example ## @@ -93,6 +103,8 @@ In general we propose two kinds of doc-strings ``` ## Doc-string referencing the file expression + +Uses `#|` instead of `##` to reference the whole file. Must be at top of the file. ```nix #| From 402f0653571f945677bcae2fb76f06e10ea42b3c Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 30 Mar 2023 21:33:37 +0200 Subject: [PATCH 015/110] Update rfcs/0145-doc-strings.md Co-authored-by: Anderson Torres --- rfcs/0145-doc-strings.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 556d34aff..897f77e6e 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -454,3 +454,8 @@ The following is an idea for a problem that will arise if tools try to track doc - `NixOS/nix` should implement native support for doc-strings. That way our users don't have to rely on nixpkgs or external tools. Those tools can still exist and provide more custom functionality, but it should be natively possible to document your nix expressions. - Every existing and future tool can implement against this RFC and rely on it. + +# References + +- [Rustdoc](https://doc.rust-lang.org/rustdoc/how-to-write-documentation.html) +- [Nixdoc](https://github.com/nix-community/nixdoc) From fa2cb38058f2fa6f2b6f50269320098a8f2f9459 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sat, 1 Apr 2023 00:29:26 +0200 Subject: [PATCH 016/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 897f77e6e..a346bfec5 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -150,7 +150,7 @@ Proposed Solution (0,0) => `##` For docstring body and markdown headings `# H1` ### Format Rules -- [F100] - doc-string are all comments. That start with `##` or `#!` e.g. `## {content}` +- [F100] - doc-string are all comments. That start with `##` or `#|` e.g. `## {content}` This RFC is a significant change to the existing documentation convention. This is because it is better to do it right when always being downward compatible holds you back. We created a pro-con list in the [alternatives](#alternatives) section below. From 2fdb3c29d18daed697149fedf9f0b64502b264aa Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sat, 1 Apr 2023 11:40:27 +0200 Subject: [PATCH 017/110] Update rfcs/0145-doc-strings.md Co-authored-by: Anderson Torres --- rfcs/0145-doc-strings.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index a346bfec5..1ed6bc5a8 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -13,9 +13,6 @@ related-issues: (will contain links to implementation PRs) Standard for Docstring format -Wikipedia states: - -> In programming, a docstring is a string literal specified in source code that is used, like a comment, to document a specific segment of code. Unlike conventional source code comments, or even specifically formatted comments like docblocks, docstrings are not stripped from the source tree when it is parsed and are retained throughout the runtime of the program. This allows the programmer to inspect these comments at run time, for instance as an interactive help system, or as metadata. # Motivation [motivation]: #motivation From 805a8e1f00dbe1e5e73dba7e5b3d1fdb380eb8ef Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sat, 1 Apr 2023 11:42:46 +0200 Subject: [PATCH 018/110] Update rfcs/0145-doc-strings.md Co-authored-by: Anderson Torres --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 1ed6bc5a8..ad20ea8e5 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -11,7 +11,7 @@ related-issues: (will contain links to implementation PRs) # Summary [summary]: #summary -Standard for Docstring format +Propose a standard format for Docblocks # Motivation From c9d19a60a533faa02ed88c74d4fdb4d867f1bf7f Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sat, 1 Apr 2023 11:43:05 +0200 Subject: [PATCH 019/110] Update rfcs/0145-doc-strings.md Co-authored-by: Anderson Torres --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index ad20ea8e5..75cadaefd 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -1,5 +1,5 @@ --- -feature: doc-strings +feature: docblock-standard start-date: 2023-03-27 author: hsjobeki co-authors: (find a buddy later to help out with the RFC) From e677af50749337f400b826c4e8ab669c6b4b55f9 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 2 Apr 2023 13:58:33 +0200 Subject: [PATCH 020/110] Restructuring - Place the 'Alternatives' first (make the discussion more apparent) - Motivation rework --- rfcs/0145-doc-strings.md | 444 ++++++++++++++++++++++----------------- 1 file changed, 253 insertions(+), 191 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 75cadaefd..52b0ab2d4 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -17,69 +17,110 @@ Propose a standard format for Docblocks # Motivation [motivation]: #motivation -This RFC aims to improve the consistency of in-code documentation. (aka Doc-strings) +This RFC aims to improve the consistency of in-code documentation. (aka Doc-strings/Doc-blocks) -The community offers tools and methods (such as nixdoc, nix-doc, etc.) to write and process in-code documentation for functions and other code-related atomic expressions. -We are currently utilizing this functionality to build a subset of documentation for nix functions. (e.g., nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) +> Doc-strings and Doc-blocks aren't technicially indentical. But for simplicity the phrase `Doc-string` is used througout this document. -Other tools that work directly with the nix AST and comments: +## Current State -- [noogle](https://noogle.dev) - Nix API search engine. It allows you to search functions and other expressions. -- [nix-doc](https://github.com/lf-/nix-doc) - A Nix developer tool leveraging the rnix Nix parser for intelligent documentation search and tags generation -- [manix](https://github.com/mlvzk/manix) - A fast CLI documentation searcher for Nix. +We are currently utilizing the doc-string functionality to build a subset of documentation for nix functions. (e.g., nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) -All those tools would profit from the proposed changes. +Currently there are many inconsistently written comments that document specific parts of nixpkgs and other nix-frameworks. (see [references](#references)) +We use some of them to generate documentation automatically. (e.g., nixpkgs/lib via [nixdoc](https://github.com/nix-community/nixdoc) -So the format of that __doc-strings__ needs to be better documented and standardized. +This solution requires a lot of handwork to setup, more specifically *nixdoc* is a custom tool, that works only for that purpose. -This RFC aims to achieve consistency for doc-strings and allows for differentiation between regular comments and doc-strings. +Here is an example how the format used in *nixdoc* works: -We could envision native nix support, a documentation team, or community-driven solutions for automatically generating documentation from them. +```nix +#attrsets.nix (simplified) -More specifically, we envision the following possible features: (Long term) +{ lib }: +# Operations on attribute sets. -- Automatically build documentation - At most atomic things that directly relate to one specific piece of code (e.g., mkDerivation, make-disk-image, lib-functions, etc.) -- Hover information in IDEs -- Autocomplete in IDEs -- Detect broken code/misuse of specific functionality. +let +# ... +in +{ + /* + Example: + + Type: + + */ + AttrFunc = + # + arg1: + # + arg2: + + # ... implementation +} +``` -__Current issue__ +## Drawbacks -> The existing doc-strings heavily depend on a tool called `nixdoc` not vice versa. -> -> Instead, we want to provide a common standard that every nix user can refer to. +##### Unspecified format -Everything until now is just a draft; if you can provide better ideas, e.g. using different formats or syntax, please let me know. +The format for writing documentation-strings in general is **not specified**. The only place where it is applied is: *nixpkgs/lib/** -## Example of the current format +*nixdoc* only applies to places in nixpkgs/lib. But extending the scope of *nixdoc* is not the primary goal. Instead we should find formal rules how to write *doc-strings*. Tools like *nixdoc* can then implement against this rfc. Instead of the format relying on nixdoc implementation details. -```nix -/* - - Type: - -*/ -expr = -# Describes 'arg' -arg: -# Describes 'foo' -foo: -``` +##### Only specific places -> This RFC aims for general rules for doc-strings. -> Features like: "what different sections exist" and if they might have complex rules (e.g., type: syntax) are not specified. -> -> Providing a formal skeleton where the nix community can extend sections. +The placement of those comments requires to be exactly at the attribute-set that contains the function declarations. Which is not usable for general purpose documentation-strings. -## Proposed solution +e.g., -In general we propose two kinds of doc-strings +- file that directly exports the lib-function without wrapping it in an attribute-set. +- file that exports a constant +- files outside of lib are not / cannot be rendered -## Doc-string referencing the subsequent expression - -Uses `##` to create visual blocks and draw attention to documentation. +##### Differentiate from regular comments + +The format doesnt allow any distinction between doc-strings or regular comments. + +Having a distinction would allow to + +1. Find all comments that are part of the documentation +2. Render them into the documentation format +3. Connect the documentation to exact places in the nix-code. This is already done, but only for nixpkgs/lib. + +##### + +##### References + +###### nixpkgs - Dosctrings examples + +- [lib/attrsets](https://github.com/NixOS/nixpkgs/blob/master/lib/attrsets.nix) +- [trivial-builders](https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/trivial-builders.nix) +- [stdenv/mkDerivation](https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix) +- [nixos/lib/make-disk-image](https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-disk-image.nix) +- more ... + +###### frameworks + +- [dream2nix/utils](https://github.com/nix-community/dream2nix/blob/main/src/utils/config.nix) +- [dream2nix/templates/builder](https://github.com/nix-community/dream2nix/blob/main/src/templates/builders/default.nix) +- more ... + +###### Other tools + +Other tools that work directly with the nix AST and comments: + +- [noogle](https://noogle.dev) - Nix API search engine. It allows you to search functions and other expressions. +- [nix-doc](https://github.com/lf-/nix-doc) - A Nix developer tool leveraging the rnix Nix parser for intelligent documentation search and tags generation +- [manix](https://github.com/mlvzk/manix) - A fast CLI documentation searcher for Nix. + +## Proposed Improvement - Doc-blocks + +In general I thought we do need two kinds of doc-strings: + +> The format is not final yet. Discussion may still ongoing. + +#### A doc-string referencing the subsequent expression + +Example: ```nix # somefile.nix @@ -99,9 +140,9 @@ Uses `##` to create visual blocks and draw attention to documentation. } ``` -## Doc-string referencing the file expression +#### A Doc-string referencing the file expression -Uses `#|` instead of `##` to reference the whole file. Must be at top of the file. +Example: Uses `#|` instead of `##` to reference the whole file. Must be at top of the file. ```nix #| @@ -120,159 +161,37 @@ Uses `#|` instead of `##` to reference the whole file. Must be at top of the fil } ``` -We considered this possible solution because the following reasons: - -- Good visual distinction between doc-strings and comments -- Clear rules and structure -- Saves vertical space -- Doesn't need termination (*/) -- Clear indentation rules. Following the __markdown__ convention. - -# Detailed design -[design]: #detailed-design - -The following abstract rules describe how to write doc-strings. - -> I would be pleased if you comment about whether we should use `## {content}` or `/** {content} */` -> I did write this RFC tailored towards `##` but using `/** */` is still an open discussion. - -We must find one solution out of the following: - -| | 0 `##` | 1 `/** */` | -|---|---|---| -| 0 `# {Keyword}` | `## # Example` | `/** # Example */` | -| 1 `@{Keyword}:` | `# @Example:` |`/** @Example: */` | - -Proposed Solution (0,0) => `##` For docstring body and markdown headings `# H1` - -### Format Rules - -- [F100] - doc-string are all comments. That start with `##` or `#|` e.g. `## {content}` - -This RFC is a significant change to the existing documentation convention. This is because it is better to do it right when always being downward compatible holds you back. We created a pro-con list in the [alternatives](#alternatives) section below. - -We finally need to distinguish between regular comments and doc strings. We found this format to be the most distinguishable. - -- [F200] - Doc-strings always document / relate to an expression. - -- [F201] - Doc-strings starting with `##` relate to the expression in the following line / or, more precisely, to the next node in the AST. (Details follow, as this might be non-trivial) -- [F202] - Doc-strings that are at the top of a file and that start with `#|` describe the expression exported from the whole file. (Previous node in AST) - -In comparison, rustdoc uses `//!`. But using `#!` is considered a bad idea, as it can be confused with identical bash shebangs `#!`. - -> This is still being determined. If you have any ideas, let us know in the comments. - -Example of a comment referring to the whole file: - -```nix -#| -#| -#| # Example -#| -#| # Type -#| The Type of the expression returned by the file -{ - id = x: x -} -``` - -- [F300] - The docstring is continued in the following line if it also starts with `##` / `#|`. Leading whitespace is allowed. - -Example: docstring continuation - -```nix -## Doc-string A -## .... -## This block has no expression in the next line. -## Therefore it doesn't have any effect -## .... -## Doc-string A - - -## Doc-string B -## -- this block documents the purpose of '1' -## Doc-string B -1 -``` - -### Structural Rules - -- [S010] - The content of a doc-string is Markdown. - -It allows for intuitive usage without knowledge of complex syntax rules. - -- [S011] - predefined heading `keywords` may start a section. -- [S021] - Content before the first [optional] section is called `description`. - -This allows for quick writing without the need to use sections. - -- [S022] - Headings H1 are reserved markdown headings. Which are specified in [this list](#keywords). Users are allowed to only use H2 (or higher) headings for their free use. - -H1 headings start a section to keep it extendable in the future. Users do not choose them freely, so we keep track of all allowed H1 headings. - -- [S012] - Every [optional] section started by an H1 heading is continued until the next heading starts. To the very end of the comment, otherwise. -- [S014] - Every section defines its own rules. They must be compatible with the formal requirements of doc-strings (this RFC) that can override formal rules locally. (e.g., disable Markdown, use custom syntax, etc.) -- [S017] - Only the H1-sections (`Keywords`) described in [this list](#keywords) are valid. - -- [S018] - In case of extension, every new section `Keyword` must first be added to this RFC. -- [S030] - If sections follow complex logic, it is embraced to specify that logic in a separate sub-RFC. -- [S040] - Usage of the described sections is OPTIONAL. -- ... more tbd. - -## Keywords -[keywords]: #keywords - -We wanted to keep the list of initial keywords short. So by the time this RFC focuses on the formal aspects of doc-strings first. More keywords and features for them can be added later on. - -| Keyword | Description | Note | -| --- | --- | --- | -| `Example` | Starts the Example-block. Often contains comprehensive code examples | | -| `Type` | Start the Type-block; it is any free text | Syntax may eventually be specified in the future. [preview](https://typednix.dev). | - -## Why change the existing section specifiers? - -First of all: There are no actual block specifiers within nix or nixpkgs. The existing blocks heavily depend on a tool called `nixdoc` and not vice versa. - --> See [github:nix-community/nixdoc](https://github.com/nix-community/nixdoc) - -> `nixdoc` MUST be changed to support this RFC. (See [Future work](#future-work)) - -The sequence `Example:` has some drawbacks when it comes to syntax: - -1. It is possible that this sequence occurs in a natural text without the intention to start a new doc-string section. -2. It doesn't visually stand out. -3. It is bad that the line needs to start with `Example:` to be valid syntax. Although it is a good practice while writing comments. This shouldn't be syntactically required. > (`nixdoc` requires it). - -## Interactions - -Why doc-strings are valuable +# Alternatives +[alternatives]: #alternatives -Doc-strings can be attached to AST nodes without affecting the actual compile-, evaluation- or build-time because they are just comments. Specialized tools can handle those comments and create static documentation from them. Also, integration with LSP is possible. +In general we need: -Many files within nixpkgs contain detailed comments we cannot currently use. -An example: [make-disk-image.nix](https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-disk-image.nix) +1. General format for doc-strings. +2. Format for headings and the allowed content. -Following this RFC means refactoring for existing comments, but it also means that we can finally use all comments that were intended to be doc-strings +> It would be nice if this can be close to the markdown format. +> Because markdown is considered the easiest and most accepted format +> for writing and rednering documentation. -# Drawbacks -[drawbacks]: #drawbacks -- Changes the existing comments inside the code base. +#### Find a suitable decision in - the follwing matrix -This could mostly be automated. (e.g via codemod) +| General / Section | 0 `##` | 1 `/** */` | 2 `other` | +|---|---|---|---| +| 0 `# {Keyword}` | `## # Example` | `/** # Example */` | `?` | +| 1 `@{Keyword}:` | `# @Example:` |`/** @Example: */` | `?` | +| 2 `other` | `?` | `?` | `?` | -Also, this affects only the `lib` folder and a few other places that are currently used to build the documentation. - -# Alternatives -[alternatives]: #alternatives +It is also possible to not implement this rfc: - By not implementing this feature, nix loses the ability for tool-generated documentation. - - Documentation within code will remain unstable / only determined by nixdoc. -## Pros-Cons for the different formats +## The different formats considered -### `##` inspired from rust's `///` +### `##` inspired from rust's `///` + +with `Markdown` Headings Example: @@ -302,9 +221,9 @@ Example: | Works nicely with Markdown content as Indentation is visually more clear | Many `#` symbols might be confusing | | | Starting every line with `##` creates visual conflicts with markdown headings `# {Heading}` | -### `/** */` inspired by the current multiline strings +### `/** */` inspired by the current multiline strings -`/** */` In comparison arguments for using `/** */` together with `@{keyword}:` to start sections +With `@{keyword}:` Headings Example: @@ -355,7 +274,7 @@ Although this has already sneaked into some nix comments. This format is not con 4. Pro: Indentation within is clear. 5. Most nix users cannot identify with java or javascript. They like predictable behavior. -## Pros-Cons for section headings +## Section headings considered ### Markdown headings @@ -413,6 +332,139 @@ Example: There is the idea from python that doc-strings are just strings, not even special ones. Comments will be docstrings if they follow specific placement rules. However, we thought this was a bad idea to follow. Such complex placement rules require the users to understand where those places are; with nix syntax, this is slightly more complex than with python. Because we don't have keywords such as `class MyClass():` or `def function():` where placement would be obvious +# Detailed design +[design]: #detailed-design + +## Proposed Solution (0,0) => `##` For docstring body and markdown headings `# H1` + +The following abstract rules describe how to write doc-strings. + +> I would be pleased if you comment about whether we should use `## {content}` or `/** {content} */` +> I did write this RFC tailored towards `##` but using `/** */` is still an open discussion. + +### Format Rules (When choosing (0,0) ) + +- [F100] - doc-string are all comments. That start with `##` or `#|` e.g. `## {content}` + +This RFC is a significant change to the existing documentation convention. This is because it is better to do it right when always being downward compatible holds you back. We created a pro-con list in the [alternatives](#alternatives) section below. + +We finally need to distinguish between regular comments and doc strings. We found this format to be the most distinguishable. + +- [F200] - Doc-strings always document / relate to an expression. + +- [F201] - Doc-strings starting with `##` relate to the expression in the following line / or, more precisely, to the next node in the AST. (Details follow, as this might be non-trivial) +- [F202] - Doc-strings that are at the top of a file and that start with `#|` describe the expression exported from the whole file. (Previous node in AST) + +In comparison, rustdoc uses `//!`. But using `#!` is considered a bad idea, as it can be confused with identical bash shebangs `#!`. + +The `|` (pipe) is also avaiable in symbols used for the nix grammar. + +> This is still being determined. If you have any ideas, let us know in the comments. + +Example of a comment referring to the whole file: + +```nix +#| +#| +#| # Example +#| +#| # Type +#| The Type of the expression returned by the file +{ + id = x: x +} +``` + +- [F300] - The docstring is continued in the following line if it also starts with `##` / `#|`. Leading whitespace is allowed. + +Example: docstring continuation + +```nix +## Doc-string A +## .... +## This block has no expression in the next line. +## Therefore it doesn't have any effect +## .... +## Doc-string A + + +## Doc-string B +## -- this block documents the purpose of '1' +## Doc-string B +1 +``` + +### Structural Rules + +- [S010] - The content of a doc-string is Markdown. + +It allows for intuitive usage without knowledge of complex syntax rules. + +- [S011] - predefined heading `keywords` may start a section. +- [S021] - Content before the first [optional] section is called `description`. + +This allows for quick writing without the need to use sections. + +- [S022] - Headings H1 are reserved markdown headings. Which are specified in [this list](#keywords). Users are allowed to only use H2 (or higher) headings for their free use. + +H1 headings start a section to keep it extendable in the future. Users are not allowed to choose them freely, so we keep track of all allowed H1 headings. + +This may also be checked from the doc-tool that may evolve from this rfc (e.g. future versions of nixdoc) + +- [S012] - Every [optional] section started by an H1 heading is continued until the next heading starts. To the very end of the comment, otherwise. +- [S014] - Every section may define its own rules. They must be compatible with the formal requirements of doc-strings (this RFC) that can override formal rules locally. (e.g., disable Markdown, use custom syntax, etc.) +- [S017] - Only the H1-sections (`Keywords`) described in [this list](#keywords) are valid. + +- [S018] - In case of [future] extensions, every new section `Keyword` must first be added to this RFC. +- [S030] - If sections follow complex logic, it is embraced to specify that logic in a separate sub-RFC. +- [S040] - Usage of the described sections is OPTIONAL. +- ... more tbd. + +## Keywords +[keywords]: #keywords + +We wanted to keep the list of initial keywords short. So by the time this RFC focuses on the formal aspects of doc-strings first. More keywords and features for them can be added later on. + +| Keyword | Description | Note | +| --- | --- | --- | +| `Example` | Starts the Example-block. Often contains comprehensive code examples | | +| `Type` | Start the Type-block; it is any free text | Syntax may eventually be specified in the future. [preview](https://typednix.dev). | + +## Why change the existing section specifiers? + +First of all: There are no actual block specifiers within nix or nixpkgs. The existing blocks heavily depend on a tool called `nixdoc` and not vice versa. + +-> See [github:nix-community/nixdoc](https://github.com/nix-community/nixdoc) + +> `nixdoc` MUST be changed to support this RFC. (See [Future work](#future-work)) + +The sequence `Example:` has some drawbacks when it comes to syntax: + +1. It is possible that this sequence occurs in a natural text without the intention to start a new doc-string section. +2. It doesn't visually stand out. +3. It is bad that the line needs to start with `Example:` to be valid syntax. Although it is a good practice while writing comments. This shouldn't be syntactically required. > (`nixdoc` requires it). +4. It neither follows the `@param` (c/c++/java,...) convention nor the markdown headings convention (rust), instead is nixdoc-home-cooked. + +## Interactions + +Doc-strings can be attached to AST nodes without affecting the actual compile-, evaluation- or build-time because they are just comments. Specialized tools can handle those comments and create static documentation from them. Also, integration with LSP is possible. + +Many files within nixpkgs contain detailed comments we cannot currently use. +An example: [make-disk-image.nix](https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-disk-image.nix) + +Following this RFC means refactoring for existing comments, but it also means that **we can finally use all comments (automated!) that were intended to be doc-strings** + +# Drawbacks +[drawbacks]: #drawbacks + +Drawbacks of this rfc. + +- Changes the existing comments inside the code base. + +This could mostly be automated. (e.g via codemod) + +Also, this affects only the `lib` folder and a few other places that are currently used to build the documentation. + # Unresolved questions [unresolved]: #unresolved-questions @@ -456,3 +508,13 @@ The following is an idea for a problem that will arise if tools try to track doc - [Rustdoc](https://doc.rust-lang.org/rustdoc/how-to-write-documentation.html) - [Nixdoc](https://github.com/nix-community/nixdoc) + +## + +We considered this possible solution because the following reasons: + +- Good visual distinction between doc-strings and comments +- Clear rules and structure +- Saves vertical space +- Doesn't need termination (*/) +- Clear indentation rules. Following the __markdown__ convention. From 84b29fbaff372431012eed649a1590b01389631c Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 2 Apr 2023 13:59:53 +0200 Subject: [PATCH 021/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 52b0ab2d4..470db4a6f 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -23,7 +23,7 @@ This RFC aims to improve the consistency of in-code documentation. (aka Doc-stri ## Current State -We are currently utilizing the doc-string functionality to build a subset of documentation for nix functions. (e.g., nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) +We are currently utilizing a `doc-string`-like functionality to build a subset of documentation for nix functions. (e.g., nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) Currently there are many inconsistently written comments that document specific parts of nixpkgs and other nix-frameworks. (see [references](#references)) We use some of them to generate documentation automatically. (e.g., nixpkgs/lib via [nixdoc](https://github.com/nix-community/nixdoc) From d5e3083709aedd830d213d78f6008bdf4d321aca Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 2 Apr 2023 21:50:10 +0200 Subject: [PATCH 022/110] fix some minor grammar mistakes --- rfcs/0145-doc-strings.md | 92 ++++++++++++++++++++-------------------- 1 file changed, 45 insertions(+), 47 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 470db4a6f..25d3c224f 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -11,7 +11,7 @@ related-issues: (will contain links to implementation PRs) # Summary [summary]: #summary -Propose a standard format for Docblocks +Propose a standard format for Docblocks. # Motivation @@ -19,21 +19,21 @@ Propose a standard format for Docblocks This RFC aims to improve the consistency of in-code documentation. (aka Doc-strings/Doc-blocks) -> Doc-strings and Doc-blocks aren't technicially indentical. But for simplicity the phrase `Doc-string` is used througout this document. +> Doc-strings and Doc-blocks are technically different. But for simplicity, the phrase `Doc-string` is used in this document for clarity. ## Current State -We are currently utilizing a `doc-string`-like functionality to build a subset of documentation for nix functions. (e.g., nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) +We currently utilize a `doc-string`-like functionality to build a subset of documentation for nix functions. (e.g., nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) -Currently there are many inconsistently written comments that document specific parts of nixpkgs and other nix-frameworks. (see [references](#references)) +Many inconsistently written comments document specific parts of nixpkgs and other nix-frameworks. (see [references](#references)) We use some of them to generate documentation automatically. (e.g., nixpkgs/lib via [nixdoc](https://github.com/nix-community/nixdoc) -This solution requires a lot of handwork to setup, more specifically *nixdoc* is a custom tool, that works only for that purpose. +This solution requires a lot of handworks; more specifically, *nixdoc* is a custom tool that works only for that purpose. -Here is an example how the format used in *nixdoc* works: +Here is an example of how the format used in *nixdoc* works: ```nix -#attrsets.nix (simplified) +#attrsets. nix (simplified) { lib }: # Operations on attribute sets. @@ -62,31 +62,29 @@ in ##### Unspecified format -The format for writing documentation-strings in general is **not specified**. The only place where it is applied is: *nixpkgs/lib/** +In general, the format for writing documentation strings is **not specified**. The only place where it is applied is: *nixpkgs/lib/** -*nixdoc* only applies to places in nixpkgs/lib. But extending the scope of *nixdoc* is not the primary goal. Instead we should find formal rules how to write *doc-strings*. Tools like *nixdoc* can then implement against this rfc. Instead of the format relying on nixdoc implementation details. +*nixdoc* only applies to places in nixpkgs/lib. But extending the scope of *nixdoc* is not the primary goal. Instead, we should find formal rules for writing *doc-strings*. Tools like *nixdoc* can then implement against this RFC instead of the format relying on nixdoc implementation details. ##### Only specific places -The placement of those comments requires to be exactly at the attribute-set that contains the function declarations. Which is not usable for general purpose documentation-strings. +The placement of those comments requires precisely looking at the attribute set containing the function declarations, which is not usable for general-purpose documentation strings. e.g., -- file that directly exports the lib-function without wrapping it in an attribute-set. +- file that directly exports the lib-function without wrapping it in an attribute set. - file that exports a constant - files outside of lib are not / cannot be rendered ##### Differentiate from regular comments -The format doesnt allow any distinction between doc-strings or regular comments. +The format doesn't allow any distinction between doc-strings or regular comments. -Having a distinction would allow to +Having a distinction would allow us to 1. Find all comments that are part of the documentation -2. Render them into the documentation format -3. Connect the documentation to exact places in the nix-code. This is already done, but only for nixpkgs/lib. - -##### +2. Render them in the documentation format +3. Connect the documentation to the exact places in the nix code. This is already done, but only for nixpkgs/lib. ##### References @@ -96,13 +94,13 @@ Having a distinction would allow to - [trivial-builders](https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/trivial-builders.nix) - [stdenv/mkDerivation](https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix) - [nixos/lib/make-disk-image](https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-disk-image.nix) -- more ... +- more... ###### frameworks - [dream2nix/utils](https://github.com/nix-community/dream2nix/blob/main/src/utils/config.nix) - [dream2nix/templates/builder](https://github.com/nix-community/dream2nix/blob/main/src/templates/builders/default.nix) -- more ... +- more... ###### Other tools @@ -110,13 +108,13 @@ Other tools that work directly with the nix AST and comments: - [noogle](https://noogle.dev) - Nix API search engine. It allows you to search functions and other expressions. - [nix-doc](https://github.com/lf-/nix-doc) - A Nix developer tool leveraging the rnix Nix parser for intelligent documentation search and tags generation -- [manix](https://github.com/mlvzk/manix) - A fast CLI documentation searcher for Nix. +- [manix](https://github.com/mlvzk/manix) - A fast CLI documentation searcher for nix. ## Proposed Improvement - Doc-blocks -In general I thought we do need two kinds of doc-strings: +In general, I thought we do need two kinds of doc-strings: -> The format is not final yet. Discussion may still ongoing. +> The format is not final yet. Discussion may still be ongoing. #### A doc-string referencing the subsequent expression @@ -142,7 +140,7 @@ Example: #### A Doc-string referencing the file expression -Example: Uses `#|` instead of `##` to reference the whole file. Must be at top of the file. +Example: Uses `#|` instead of `##` to reference the whole file. It must be at the top of the file. ```nix #| @@ -164,17 +162,17 @@ Example: Uses `#|` instead of `##` to reference the whole file. Must be at top o # Alternatives [alternatives]: #alternatives -In general we need: +In general, we need the following: 1. General format for doc-strings. 2. Format for headings and the allowed content. -> It would be nice if this can be close to the markdown format. -> Because markdown is considered the easiest and most accepted format -> for writing and rednering documentation. +> It would be nice if this could be close to the markdown format. +> Markdown is the most straightforward and most accepted format +> for writing and rendering documentation. -#### Find a suitable decision in - the follwing matrix +#### Find a suitable decision in - the following matrix | General / Section | 0 `##` | 1 `/** */` | 2 `other` | |---|---|---|---| @@ -182,7 +180,7 @@ In general we need: | 1 `@{Keyword}:` | `# @Example:` |`/** @Example: */` | `?` | | 2 `other` | `?` | `?` | `?` | -It is also possible to not implement this rfc: +It is also possible to not implement this RFC: - By not implementing this feature, nix loses the ability for tool-generated documentation. - Documentation within code will remain unstable / only determined by nixdoc. @@ -266,13 +264,13 @@ Javadoc style } ``` -Although this has already sneaked into some nix comments. This format is not considered best practice for a variety of good reasons. +Although this has already sneaked into some nix comments, this format is not considered best practice for various reasons. 1. Essentially combines the cons of both worlds. 2. It Takes up more space and needs autocompletion to fill missing `*` beginnings when extended. 3. Starting every line with `*` creates visual conflicts with the markdown bullet list also starting with `*`. 4. Pro: Indentation within is clear. -5. Most nix users cannot identify with java or javascript. They like predictable behavior. +5. Most nix users cannot identify with java or javascript and like predictable behavior. ## Section headings considered @@ -326,7 +324,7 @@ Example: | Visually stands out | Is new syntax. Where Markdown could be more intuitive. Doc-strings already are Markdown. So why not use markdown | | Follows more closely the current convention | | | Needs less vertical space | | -| doesn't need newlines, everything could be even within a single line, allowing compression (may not be needed ?) | | +| doesn't need newlines; everything could be even within a single line, allowing compression (may not be needed ?) | | ## Alternative approach - just comments @@ -346,7 +344,7 @@ The following abstract rules describe how to write doc-strings. - [F100] - doc-string are all comments. That start with `##` or `#|` e.g. `## {content}` -This RFC is a significant change to the existing documentation convention. This is because it is better to do it right when always being downward compatible holds you back. We created a pro-con list in the [alternatives](#alternatives) section below. +This RFC is a significant change to the existing documentation convention. It is better to do it right when always being downward compatible holds you back. We finally need to distinguish between regular comments and doc strings. We found this format to be the most distinguishable. @@ -357,7 +355,7 @@ We finally need to distinguish between regular comments and doc strings. We foun In comparison, rustdoc uses `//!`. But using `#!` is considered a bad idea, as it can be confused with identical bash shebangs `#!`. -The `|` (pipe) is also avaiable in symbols used for the nix grammar. +The `|` (pipe) is also available in symbols used for the nix grammar. > This is still being determined. If you have any ideas, let us know in the comments. @@ -382,14 +380,14 @@ Example: docstring continuation ```nix ## Doc-string A ## .... -## This block has no expression in the next line. -## Therefore it doesn't have any effect +## This block has no expression in the following line. +## Therefore, it doesn't have any effect ## .... ## Doc-string A ## Doc-string B -## -- this block documents the purpose of '1' +## -- This block documents the purpose of '1' ## Doc-string B 1 ``` @@ -409,7 +407,7 @@ This allows for quick writing without the need to use sections. H1 headings start a section to keep it extendable in the future. Users are not allowed to choose them freely, so we keep track of all allowed H1 headings. -This may also be checked from the doc-tool that may evolve from this rfc (e.g. future versions of nixdoc) +This may also be checked from the doc-tool that may evolve from this RFC (e.g. future versions of nixdoc) - [S012] - Every [optional] section started by an H1 heading is continued until the next heading starts. To the very end of the comment, otherwise. - [S014] - Every section may define its own rules. They must be compatible with the formal requirements of doc-strings (this RFC) that can override formal rules locally. (e.g., disable Markdown, use custom syntax, etc.) @@ -418,12 +416,12 @@ This may also be checked from the doc-tool that may evolve from this rfc (e.g. f - [S018] - In case of [future] extensions, every new section `Keyword` must first be added to this RFC. - [S030] - If sections follow complex logic, it is embraced to specify that logic in a separate sub-RFC. - [S040] - Usage of the described sections is OPTIONAL. -- ... more tbd. +- more tbd. ## Keywords [keywords]: #keywords -We wanted to keep the list of initial keywords short. So by the time this RFC focuses on the formal aspects of doc-strings first. More keywords and features for them can be added later on. +We wanted to keep the list of initial keywords short. So by the time this RFC focuses on the formal aspects of doc-strings first. More keywords and features for them may be added later on. | Keyword | Description | Note | | --- | --- | --- | @@ -432,7 +430,7 @@ We wanted to keep the list of initial keywords short. So by the time this RFC fo ## Why change the existing section specifiers? -First of all: There are no actual block specifiers within nix or nixpkgs. The existing blocks heavily depend on a tool called `nixdoc` and not vice versa. +First, there are no actual block specifiers within nix or nixpkgs. The existing blocks heavily depend on a tool called `nixdoc` and not vice versa. -> See [github:nix-community/nixdoc](https://github.com/nix-community/nixdoc) @@ -442,8 +440,8 @@ The sequence `Example:` has some drawbacks when it comes to syntax: 1. It is possible that this sequence occurs in a natural text without the intention to start a new doc-string section. 2. It doesn't visually stand out. -3. It is bad that the line needs to start with `Example:` to be valid syntax. Although it is a good practice while writing comments. This shouldn't be syntactically required. > (`nixdoc` requires it). -4. It neither follows the `@param` (c/c++/java,...) convention nor the markdown headings convention (rust), instead is nixdoc-home-cooked. +3. It is terrible that the line needs to start with `Example:` to be valid syntax. However, it is a good practice while writing comments; it should be optional. +4. It neither follows the `@param` (c/c++/java,...) convention nor the markdown headings convention (rust); instead is nixdoc-home-cooked. ## Interactions @@ -472,7 +470,7 @@ Also, this affects only the `lib` folder and a few other places that are current - Will `nix` itself implement native support like in rust -> `cargo doc` -- How can a tool keep the connection from where a docstring was defined and where the attribute was exposed (lib/default.nix exposes mapAttrs which is defined at lib/attrsets.nix) +- How can a tool keep the connection from where a docstring was defined and where the attribute was exposed (lib/default. nix exposes mapAttrs which is defined at lib/attrsets. nix) - There are more complicated things. -> Answer: A Tool might be able to keep track of a percentage of expressions, and sometimes it may be very hard or impossible. For that case, the doc-string can offer a dedicated Keyword to override the scope. @@ -496,11 +494,11 @@ The following is an idea for a problem that will arise if tools try to track doc # Future work [future]: #future-work -- When extending nixdoc and/or writing dedicated parsers the following persons can assist: [@hsjobeki] +- When extending nixdoc or writing dedicated parsers, the following persons can assist: [@hsjobeki] -- There is an RFC under construction, that specifies the used syntax within the `Type`-Block. It depends on this RFC, as this RFC is the groundwork to provide a standardized field where additional rules can apply. Core-Team: [@hsjobeki] +- An RFC under construction specifies the used syntax within the `Type`-Block. It depends on this RFC, as it is the groundwork to provide a standardized field where additional rules can apply. -- `NixOS/nix` should implement native support for doc-strings. That way our users don't have to rely on nixpkgs or external tools. Those tools can still exist and provide more custom functionality, but it should be natively possible to document your nix expressions. +- `NixOS/nix` should implement native support for doc-strings so that our users don't have to rely on nixpkgs or external tools. Those tools can still exist and provide more custom functionality, but documenting your nix expressions should be natively possible. - Every existing and future tool can implement against this RFC and rely on it. From 6fc5a4168ee67f397bbe4f9804859d0486d786f5 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Tue, 4 Apr 2023 12:14:23 +0200 Subject: [PATCH 023/110] Consolidate into one big table Refactor the structure. Thanks, @infinisil, for the excellent feedback. --- rfcs/0145-doc-strings.md | 412 ++++++++++++++++++--------------------- 1 file changed, 195 insertions(+), 217 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 25d3c224f..bd7cf7b01 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -58,25 +58,25 @@ in } ``` -## Drawbacks +## Current problems -##### Unspecified format +### Unspecified format In general, the format for writing documentation strings is **not specified**. The only place where it is applied is: *nixpkgs/lib/** *nixdoc* only applies to places in nixpkgs/lib. But extending the scope of *nixdoc* is not the primary goal. Instead, we should find formal rules for writing *doc-strings*. Tools like *nixdoc* can then implement against this RFC instead of the format relying on nixdoc implementation details. -##### Only specific places +### Only specific placements work -The placement of those comments requires precisely looking at the attribute set containing the function declarations, which is not usable for general-purpose documentation strings. +The placement of those comments requires precisely commenting at the attribute set containing the function declarations, which is not usable for general-purpose documentation strings. e.g., - file that directly exports the lib-function without wrapping it in an attribute set. -- file that exports a constant -- files outside of lib are not / cannot be rendered +- file that exports a constant expression +- files outside of lib cannot be rendered due to missing conventions -##### Differentiate from regular comments +### Differentiate from regular comments The format doesn't allow any distinction between doc-strings or regular comments. @@ -86,9 +86,9 @@ Having a distinction would allow us to 2. Render them in the documentation format 3. Connect the documentation to the exact places in the nix code. This is already done, but only for nixpkgs/lib. -##### References +### References to the problems above -###### nixpkgs - Dosctrings examples +#### nixpkgs - Dosctrings examples - [lib/attrsets](https://github.com/NixOS/nixpkgs/blob/master/lib/attrsets.nix) - [trivial-builders](https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/trivial-builders.nix) @@ -96,13 +96,13 @@ Having a distinction would allow us to - [nixos/lib/make-disk-image](https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-disk-image.nix) - more... -###### frameworks +#### frameworks - [dream2nix/utils](https://github.com/nix-community/dream2nix/blob/main/src/utils/config.nix) - [dream2nix/templates/builder](https://github.com/nix-community/dream2nix/blob/main/src/templates/builders/default.nix) - more... -###### Other tools +#### Other tools Other tools that work directly with the nix AST and comments: @@ -110,13 +110,21 @@ Other tools that work directly with the nix AST and comments: - [nix-doc](https://github.com/lf-/nix-doc) - A Nix developer tool leveraging the rnix Nix parser for intelligent documentation search and tags generation - [manix](https://github.com/mlvzk/manix) - A fast CLI documentation searcher for nix. -## Proposed Improvement - Doc-blocks +# Detailed design +[design]: #detailed-design -In general, I thought we do need two kinds of doc-strings: +**Proposed Solution:** + +Use `##` For doc-string body and markdown headings `# H1` -> The format is not final yet. Discussion may still be ongoing. +The content of all doc-strings is markdown. +Following the [commonmark-specification](https://spec.commonmark.org/) -#### A doc-string referencing the subsequent expression +## Doc-blocks + +In general, I thought we do need two kinds of doc-strings: + +### A doc-string referencing the subsequent expression Example: @@ -138,7 +146,7 @@ Example: } ``` -#### A Doc-string referencing the file expression +### A Doc-string referencing the file expression Example: Uses `#|` instead of `##` to reference the whole file. It must be at the top of the file. @@ -159,199 +167,42 @@ Example: Uses `#|` instead of `##` to reference the whole file. It must be at th } ``` -# Alternatives -[alternatives]: #alternatives - -In general, we need the following: - -1. General format for doc-strings. -2. Format for headings and the allowed content. - -> It would be nice if this could be close to the markdown format. -> Markdown is the most straightforward and most accepted format -> for writing and rendering documentation. - - -#### Find a suitable decision in - the following matrix - -| General / Section | 0 `##` | 1 `/** */` | 2 `other` | -|---|---|---|---| -| 0 `# {Keyword}` | `## # Example` | `/** # Example */` | `?` | -| 1 `@{Keyword}:` | `# @Example:` |`/** @Example: */` | `?` | -| 2 `other` | `?` | `?` | `?` | - -It is also possible to not implement this RFC: - -- By not implementing this feature, nix loses the ability for tool-generated documentation. -- Documentation within code will remain unstable / only determined by nixdoc. - -## The different formats considered - -### `##` inspired from rust's `///` - -with `Markdown` Headings - -Example: - -```nix -# somefile.nix - - ## - ## - ## # Example - ## - ## - ## - ## # Type - ## - ## - mapAttrs = f: s: #... -``` - -| Pro | Con | -|---|---| -| Saves vertical space | Needs Autocompletion (Language Server) to continue the next line. Hustle otherwise to start every line by hand | -| | Changes the existing convention | -| Doesn't need termination (e.g. */) | Can break when interrupted with newlines / non-docstring line beginnings | -| Easier to read / Indentation is clear | Multiple comment tokens must be concatenated (Might be more complex) | -| Block continuation is more intuitive (With autocomplete setup properly) | | -| Uses fewer punctuations and special characters; thus is visually more clear and requires less finger spread movements for reaching / and * and @ (for sections) | | -| Works nicely with Markdown content as Indentation is visually more clear | Many `#` symbols might be confusing | -| | Starting every line with `##` creates visual conflicts with markdown headings `# {Heading}` | - -### `/** */` inspired by the current multiline strings - -With `@{keyword}:` Headings - -Example: - -```nix - /** - - - @Example: - - - - @Type: - - - */ - mapAttrs = f: s: #... -``` - -| Pro | Con | -|---|---| -| Clear termination | Takes up more vertical space | -| Doesn't change the existing convention by much | doesn't visually stand out by much (just one more `*` ) | -| Mostly stays compatible with existing implementations | Multiple blocks are not concatenated. They need to be continued | -| No configuration required (LSP for autocompletion on newlines) | | -| | Indentation needs to be clarified / more complex. | - -## Candidates not considered - -Javadoc style - -```java - /** - * A short description - * @author Stefan Schneider - * @version 1.1 - * @see https://some.url - */ - public class Product { - ... - } -``` - -Although this has already sneaked into some nix comments, this format is not considered best practice for various reasons. - -1. Essentially combines the cons of both worlds. -2. It Takes up more space and needs autocompletion to fill missing `*` beginnings when extended. -3. Starting every line with `*` creates visual conflicts with the markdown bullet list also starting with `*`. -4. Pro: Indentation within is clear. -5. Most nix users cannot identify with java or javascript and like predictable behavior. - -## Section headings considered - -### Markdown headings - -Example: - -```nix -# somefile.nix - - ## - ## - ## # Example - ## - ## - ## - ## # Type - ## - ## - mapAttrs = f: s: #... -``` - -| Pro | Con | -|---|---| -| Markdown is simple | doesn't visually distinguish from `##` starting the doc-string | -| Using headings feels natural | Users may accidentally use those headings within natural language | -| | Markdown recommends using newlines before and after headings which takes up a lot of vertical space | -| | Markdown headings do not visually stand out from lines that already started with `##` | - -### Custom headings `@{Keyword}:` - -Example: - -```nix -# somefile.nix - - ## - ## - ## @Example: - ## - ## - ## - ## @Type: - ## - ## - mapAttrs = f: s: #... -``` - -| Pro | Con | -|---|---| -| Visually stands out | Is new syntax. Where Markdown could be more intuitive. Doc-strings already are Markdown. So why not use markdown | -| Follows more closely the current convention | | -| Needs less vertical space | | -| doesn't need newlines; everything could be even within a single line, allowing compression (may not be needed ?) | | - -## Alternative approach - just comments - -There is the idea from python that doc-strings are just strings, not even special ones. Comments will be docstrings if they follow specific placement rules. However, we thought this was a bad idea to follow. Such complex placement rules require the users to understand where those places are; with nix syntax, this is slightly more complex than with python. Because we don't have keywords such as `class MyClass():` or `def function():` where placement would be obvious +The following abstract rules describe how to write doc-strings. -# Detailed design -[design]: #detailed-design +> I would be pleased if you comment about whether we should use `## {content}` or `/** {content} */` +> I did write this RFC tailored towards `##` but using `/** */` may still be an open discussion. -## Proposed Solution (0,0) => `##` For docstring body and markdown headings `# H1` +### Format Rules -The following abstract rules describe how to write doc-strings. +- doc-string are all comments. That start with `##` or `#|` e.g. `## {content}` -> I would be pleased if you comment about whether we should use `## {content}` or `/** {content} */` -> I did write this RFC tailored towards `##` but using `/** */` is still an open discussion. +This RFC is a significant change to the existing documentation convention +but we finally need to distinguish between regular comments and doc strings. We found this format to be the most distinguishable. -### Format Rules (When choosing (0,0) ) +- Doc-strings always document / relate to an expression. -- [F100] - doc-string are all comments. That start with `##` or `#|` e.g. `## {content}` +In detail: `##` relates to the next AST-Node `#|` to the previous one. However technical details for tracking names and aliases is not part of this document. -This RFC is a significant change to the existing documentation convention. It is better to do it right when always being downward compatible holds you back. +> Vision: Implement a custom evaluator that specializes in tracking references of doc-strings within the nix expression tree. This is however a technical concrete solution that may be build after this rfc is accepted. -We finally need to distinguish between regular comments and doc strings. We found this format to be the most distinguishable. +- Doc-strings starting with `##` relate to the expression in the following line / or, more precisely, to the next node in the AST. (Implementation Details are not considered yet, maybe future versions need to narrow the scope here) -- [F200] - Doc-strings always document / relate to an expression. +> I wont go into details here as this is would already be an implementation specification, but this is how i thought it could technically make sense. +> +> E.g. a "documentation generator": +> The referenced expression might be an IDENT_NODE or an expression that can be assigned to an IDENT_NODE. +> The doc-string may then be yielded from the IDENT_NODE, or from the expression that was assigned to the NODE. (Precendence is -1, which means a doc-string is always the last assignment) +> ```nix +> ## Refences the whole thing +> ## (foo bar baz) +> foo bar baz +> ``` +> Expressions would thus need a meta-wrapper that hold the necessary doc-string and preserves it during the evaluation. While rendering out the whole nix expression (recursion need to be considered e.g. in Derivation, etc.!) All references are preserved and can then be mapped into a list for each identifier in the tree. +> e.g. `foo.bar.baz` holds a refernce to the docstring that was asigned to the expression in `baz.nix` +> +> BUT this is a very concrete documentation generator, that makes eventually sense to be built after this RFC. -- [F201] - Doc-strings starting with `##` relate to the expression in the following line / or, more precisely, to the next node in the AST. (Details follow, as this might be non-trivial) -- [F202] - Doc-strings that are at the top of a file and that start with `#|` describe the expression exported from the whole file. (Previous node in AST) +- Doc-strings that are at the top of a file and that start with `#|` describe the expression exported from the whole file. (Previous node in AST) In comparison, rustdoc uses `//!`. But using `#!` is considered a bad idea, as it can be confused with identical bash shebangs `#!`. @@ -373,7 +224,7 @@ Example of a comment referring to the whole file: } ``` -- [F300] - The docstring is continued in the following line if it also starts with `##` / `#|`. Leading whitespace is allowed. +- The docstring is continued in the following line if it also starts with `##` / `#|`. Leading whitespace is allowed. Example: docstring continuation @@ -392,36 +243,36 @@ Example: docstring continuation 1 ``` -### Structural Rules - -- [S010] - The content of a doc-string is Markdown. +- The content of a doc-string is Markdown. It allows for intuitive usage without knowledge of complex syntax rules. -- [S011] - predefined heading `keywords` may start a section. -- [S021] - Content before the first [optional] section is called `description`. +- predefined heading `keywords` may start a section. +- Content before the first [optional] section is called `description`. This allows for quick writing without the need to use sections. -- [S022] - Headings H1 are reserved markdown headings. Which are specified in [this list](#keywords). Users are allowed to only use H2 (or higher) headings for their free use. +- Headings H1 are reserved markdown headings. Which are specified in [this list](#keywords). Users are allowed to only use H2 (or higher) headings for their free use. H1 headings start a section to keep it extendable in the future. Users are not allowed to choose them freely, so we keep track of all allowed H1 headings. This may also be checked from the doc-tool that may evolve from this RFC (e.g. future versions of nixdoc) -- [S012] - Every [optional] section started by an H1 heading is continued until the next heading starts. To the very end of the comment, otherwise. -- [S014] - Every section may define its own rules. They must be compatible with the formal requirements of doc-strings (this RFC) that can override formal rules locally. (e.g., disable Markdown, use custom syntax, etc.) -- [S017] - Only the H1-sections (`Keywords`) described in [this list](#keywords) are valid. +- Every [optional] section started by an H1 heading is continued until the next heading starts. To the very end of the comment, otherwise. +- Every section may define its own rules. They must be compatible with the formal requirements of doc-strings (this RFC) that can override formal rules locally. (e.g., disable Markdown, use custom syntax, etc.) +- Only the H1-sections (`Keywords`) described in [this list](#keywords) are valid. -- [S018] - In case of [future] extensions, every new section `Keyword` must first be added to this RFC. -- [S030] - If sections follow complex logic, it is embraced to specify that logic in a separate sub-RFC. -- [S040] - Usage of the described sections is OPTIONAL. +- In case of [future] extensions, every new section `Keyword` must first be added to this RFC. +- If sections follow complex logic, it is embraced to specify that logic in a separate sub-RFC. +- Usage of the described sections is OPTIONAL. - more tbd. ## Keywords [keywords]: #keywords -We wanted to keep the list of initial keywords short. So by the time this RFC focuses on the formal aspects of doc-strings first. More keywords and features for them may be added later on. +The following keywords start new markdown sections + +> I wanted to keep the list of initial keywords short. So by the time this RFC focuses on the formal aspects of doc-strings first. More keywords and features for them may be added later on. | Keyword | Description | Note | | --- | --- | --- | @@ -445,10 +296,7 @@ The sequence `Example:` has some drawbacks when it comes to syntax: ## Interactions -Doc-strings can be attached to AST nodes without affecting the actual compile-, evaluation- or build-time because they are just comments. Specialized tools can handle those comments and create static documentation from them. Also, integration with LSP is possible. - -Many files within nixpkgs contain detailed comments we cannot currently use. -An example: [make-disk-image.nix](https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-disk-image.nix) +Doc-strings can be attached to AST nodes without affecting the actual compile-, evaluation- or build-time because they are just comments. Specialized tools can handle those comments and create static documentation from them. Also, integration with LSP is possible. (See [@flokli's nix lsp-whishlist](https://hackmd.io/@geyA7YL_RyiWJO6d5TbC-g/Sy6lVrgW3) for inspirations) Following this RFC means refactoring for existing comments, but it also means that **we can finally use all comments (automated!) that were intended to be doc-strings** @@ -463,6 +311,136 @@ This could mostly be automated. (e.g via codemod) Also, this affects only the `lib` folder and a few other places that are currently used to build the documentation. + +# Alternatives +[alternatives]: #alternatives + +In general, we need the following: + +1. General format for doc-strings. +2. Format for headings and the allowed content. + +> It would be nice if this could be close to the markdown format. +> Markdown is the most straightforward and most accepted format +> for writing and rendering documentation. + +## General Formats + +| Property / Approach | `##` | `/** */` | `Javadoc` | +|---|---|---|---| +| Inspired by | Rust | Current nixpkgs.lib | C++/Java/Javascript | +| Changes the existing code by | Much | Less | Even More | +| Needs Termination | No | Yes | Yes | +| Indentation | Clear | Poor | Poor | +| Needs vertical space | No | Yes | Yes | +| Visual distinction from comments | High | Low | Intermediate | +| Needs Autocompletion (Language Server) to continue the next line. | Yes | No | Yes | +| Punctuation Variations / Amount of different special characters | Less | More | More | +| Markdown compatibility (also depends on indentation clarity) | Good, but visual conflicts with headings` #` | Poor | Intermediate | +| breaks when interrupted with newlines | Yes | No | ? | + +**Proposed format:** + +Use `##` to start a doc-string. This allows clear visual seperation from regular comments. +And provides a good compatibiliy with the strived markdown content. + +### Refactoring note: + +nixpkgs comments: + +- `##` ~4k usages (most of them are used for visual seperation e.g. `###########`) +- `#` ~20k usages +- `/*` ~6k usages +- `/**` 160 usages (most of them are completely empty ?) + +## General Headings + +| Property / Approach | `# ` | `@:` | +|---|---|---| +| Inspired by | Markdown | Doxygen | +| Changes the existing code by | Minor | Minor | +| Needs vertical space | Recommended | No | +| Visual distinction from comments | Low | High | +| Markdown compatibility | Best | None | + +**Proposed headings:** + +Use markdown headings `# `. This allows best compatibility with the aleady specified markdown/commonmark format. Allowing for easy and intuitive usage for comments. + +## Why we should do this + +- Find all comments that are part of the documentation +- Render them using markdown +- Connect the documentation to the exact places in the nix code. + +- By not implementing this feature, nix loses the ability for tool-generated documentation. +- Documentation will not defined by nixdoc, instead the community-implementation solution to the standard. + +## Examples + +This section contains examples for the different formats to visualize them and emphasize the previously discussed characteristics. + +### `##` inspired from rust's `///` + +with `Markdown` Headings + +Example: + +```nix +# somefile.nix + + ## + ## + ## # Example + ## + ## + ## + ## # Type + ## + ## + mapAttrs = f: s: #... +``` + +### `/** */` inspired by the current multiline strings + +With `@{keyword}:` Headings + +Example: + +```nix + /** + + + @Example: + + + + @Type: + + + */ + mapAttrs = f: s: #... +``` + + +## Javadoc style + +```java + /** + * A short description + * @author Stefan Schneider + * @version 1.1 + * @see https://some.url + */ + public class Product { + ... + } +``` + +## Alternative approach - just comments + +There is the idea from python that doc-strings are just strings, not even special ones. Comments will be docstrings if they follow specific placement rules. However, we thought this was a bad idea to follow. Such complex placement rules require the users to understand where those places are; with nix syntax, this is slightly more complex than with python. Because we don't have keywords such as `class MyClass():` or `def function():` where placement would be obvious + # Unresolved questions [unresolved]: #unresolved-questions From d546b39274bc257a891af729e25ad1966dbc56cd Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Tue, 4 Apr 2023 12:22:18 +0200 Subject: [PATCH 024/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index bd7cf7b01..ff1086ea9 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -19,7 +19,7 @@ Propose a standard format for Docblocks. This RFC aims to improve the consistency of in-code documentation. (aka Doc-strings/Doc-blocks) -> Doc-strings and Doc-blocks are technically different. But for simplicity, the phrase `Doc-string` is used in this document for clarity. +> Doc-strings and Doc-blocks are technically different. But for simplicity, the phrase `Doc-string` is used in this document for clarity. Because it is more common. ## Current State @@ -64,7 +64,7 @@ in In general, the format for writing documentation strings is **not specified**. The only place where it is applied is: *nixpkgs/lib/** -*nixdoc* only applies to places in nixpkgs/lib. But extending the scope of *nixdoc* is not the primary goal. Instead, we should find formal rules for writing *doc-strings*. Tools like *nixdoc* can then implement against this RFC instead of the format relying on nixdoc implementation details. +*nixdoc* only applies to places in nixpkgs/lib. But extending the scope of *nixdoc* does not work and thus is not the primary goal. Instead, we should find formal rules for writing *doc-strings*. Tools like *nixdoc* can then implement against this RFC instead of the format relying on nixdoc implementation details. ### Only specific placements work From 9f9c5d7a76bb50146ecf0f148f40d23104b315d8 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Tue, 4 Apr 2023 12:38:48 +0200 Subject: [PATCH 025/110] Extend references --- rfcs/0145-doc-strings.md | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index ff1086ea9..76903ce22 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -485,12 +485,18 @@ The following is an idea for a problem that will arise if tools try to track doc - [Rustdoc](https://doc.rust-lang.org/rustdoc/how-to-write-documentation.html) - [Nixdoc](https://github.com/nix-community/nixdoc) -## +## People that I discussed with -We considered this possible solution because the following reasons: +About doc-strings in general -- Good visual distinction between doc-strings and comments -- Clear rules and structure -- Saves vertical space -- Doesn't need termination (*/) -- Clear indentation rules. Following the __markdown__ convention. +- [@flokli](https://github.com/flokli) - one of the [tvix](https://tvl.fyi/blog/rewriting-nix) authors +- [@tazjin](https://github.com/tazjin) - Original Author of `nixdoc`, one of the `tvix` authors + +About documenation approaches on independent framworks + +- [@davHau](https://github.com/davHau) - Author of [dream2nix](https://github.com/nix-community/dream2nix) + +About defining weakly typed-interfaces for nix with doc-strings + +- [@roberth](https://github.com/roberth) - nixpkgs Architecture Team +- [@aakropotkin](https://github.com/aakropotkin/) - nixpkgs Architecture Team From 29eea3eb165877810c7ae3405faa96a4757499d0 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Tue, 4 Apr 2023 12:40:01 +0200 Subject: [PATCH 026/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 76903ce22..b361fa274 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -494,7 +494,7 @@ About doc-strings in general About documenation approaches on independent framworks -- [@davHau](https://github.com/davHau) - Author of [dream2nix](https://github.com/nix-community/dream2nix) +- [@davHau](https://github.com/davHau) - Author of [dream2nix](https://github.com/nix-community/dream2nix), (And many other nix-frameworks) About defining weakly typed-interfaces for nix with doc-strings From 18941491fc7b024beb481ac20829c79db4d96fb7 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Tue, 4 Apr 2023 12:49:26 +0200 Subject: [PATCH 027/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index b361fa274..5c9033d2a 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -315,7 +315,9 @@ Also, this affects only the `lib` folder and a few other places that are current # Alternatives [alternatives]: #alternatives -In general, we need the following: +While designing this RFC multiple alternative formats where considered. They can be found in the following section to understand the overall decisions that where made in the sections earlier. + +In general, we needed the following: 1. General format for doc-strings. 2. Format for headings and the allowed content. From b511aa1f5415d6bbcf0861a71dc0cfca61e18bda Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Tue, 4 Apr 2023 12:51:48 +0200 Subject: [PATCH 028/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 5c9033d2a..b3edf914e 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -489,6 +489,9 @@ The following is an idea for a problem that will arise if tools try to track doc ## People that I discussed with +> People mentioned here might be not yet aware of this rfc. +> I'll ping them in the next few days to make sure they are okay with beeing mentioned here. + About doc-strings in general - [@flokli](https://github.com/flokli) - one of the [tvix](https://tvl.fyi/blog/rewriting-nix) authors From a9e380b48b0a04b0b52a0a5f45c6ba0752ce8726 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Wed, 5 Apr 2023 09:30:00 +0200 Subject: [PATCH 029/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index b3edf914e..b2c4bbc6d 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -474,12 +474,41 @@ The following is an idea for a problem that will arise if tools try to track doc # Future work [future]: #future-work +## Editor support + +When starting hitting {enter} inside a doc-block the new line, should be automatically prefixed with `##` or `#|` accordingly. +This is done in rust similarily. Nix already offers a bunch of LSP's e.g. [nil](https://github.com/oxalica/nil), [rnix-lsp](https://github.com/nix-community/rnix-lsp) are the most common ones. +Those LSP's should implement the simple "line continuation" feature. (I dont know the exact name here) + +## Nixodc + +Nixdoc needs to be changed in order to differentiate between regular comment and doc-blocks. +There might be an intermediate phase of transition, where the old syntax and features is supported for a while. + - When extending nixdoc or writing dedicated parsers, the following persons can assist: [@hsjobeki] +## Documentation generators + +Generating documentation from doc-blocks is still a challenge. +If we'd like the fully automated approach, we definetly need something that can also evaluate nix expressions. +(We have such a module in `tvix` which needs to be investigated more here) + +Alternatively we can use the future nixdoc together with a static `map.json` that contains the scope for every discovery file/path in nixpkgs. + +As this second approach is much easier I propose this is how we should initially start to extend the scope. + +## More specialized section headings + +### Type + - An RFC under construction specifies the used syntax within the `Type`-Block. It depends on this RFC, as it is the groundwork to provide a standardized field where additional rules can apply. +## Native support in nix + - `NixOS/nix` should implement native support for doc-strings so that our users don't have to rely on nixpkgs or external tools. Those tools can still exist and provide more custom functionality, but documenting your nix expressions should be natively possible. +## Provide a stable and reliable format + - Every existing and future tool can implement against this RFC and rely on it. # References From 9ac29d91b68a3f3de8d7e308c8e2bc23b9b6188a Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 6 Apr 2023 11:39:34 +0200 Subject: [PATCH 030/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index b2c4bbc6d..4368fcb8c 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -25,8 +25,9 @@ This RFC aims to improve the consistency of in-code documentation. (aka Doc-stri We currently utilize a `doc-string`-like functionality to build a subset of documentation for nix functions. (e.g., nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) -Many inconsistently written comments document specific parts of nixpkgs and other nix-frameworks. (see [references](#references)) -We use some of them to generate documentation automatically. (e.g., nixpkgs/lib via [nixdoc](https://github.com/nix-community/nixdoc) +Many inconsistently written comments document specific parts of nixpkgs and other nix-frameworks (see [references-to-this](#references-to-the-problems-above)). + +We use some of them to generate documentation automatically. (e.g., nixpkgs/lib via [nixdoc](https://github.com/nix-community/nixdoc) ) This solution requires a lot of handworks; more specifically, *nixdoc* is a custom tool that works only for that purpose. @@ -511,7 +512,7 @@ As this second approach is much easier I propose this is how we should initially - Every existing and future tool can implement against this RFC and rely on it. -# References +# Related Tools - [Rustdoc](https://doc.rust-lang.org/rustdoc/how-to-write-documentation.html) - [Nixdoc](https://github.com/nix-community/nixdoc) From 1b5e5efba43057f4fe0d28452255c29ad54a11d3 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 6 Apr 2023 12:16:32 +0200 Subject: [PATCH 031/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 73 +++++++++++++++++++--------------------- 1 file changed, 34 insertions(+), 39 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 4368fcb8c..83b5e0d0d 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -1,5 +1,5 @@ --- -feature: docblock-standard +feature: doc-comment-standard start-date: 2023-03-27 author: hsjobeki co-authors: (find a buddy later to help out with the RFC) @@ -11,37 +11,32 @@ related-issues: (will contain links to implementation PRs) # Summary [summary]: #summary -Propose a standard format for Docblocks. +Propose a standard format for Doc-comments. # Motivation [motivation]: #motivation -This RFC aims to improve the consistency of in-code documentation. (aka Doc-strings/Doc-blocks) +The following are my proposed goals -> Doc-strings and Doc-blocks are technically different. But for simplicity, the phrase `Doc-string` is used in this document for clarity. Because it is more common. +- Be able to find all documentation related comments +- Be able to generate documentation from code for any nix expression. +- Avoid reading source code unnecssary. Imagine your first time reading nix source code to understand internal functions when beeing totally new to nix. ## Current State -We currently utilize a `doc-string`-like functionality to build a subset of documentation for nix functions. (e.g., nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) - +We currently utilize a `doc-comment`-like functionality to build a subset of documentation for nix functions. (e.g., nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) Many inconsistently written comments document specific parts of nixpkgs and other nix-frameworks (see [references-to-this](#references-to-the-problems-above)). We use some of them to generate documentation automatically. (e.g., nixpkgs/lib via [nixdoc](https://github.com/nix-community/nixdoc) ) This solution requires a lot of handworks; more specifically, *nixdoc* is a custom tool that works only for that purpose. -Here is an example of how the format used in *nixdoc* works: +Here is an example of the format understood by *nixdoc*: ```nix -#attrsets. nix (simplified) - -{ lib }: -# Operations on attribute sets. +# nixpkgs/lib/attrsets.nix -let -# ... -in { /* Example: @@ -61,11 +56,17 @@ in ## Current problems +### Inconsistend usage outside /lib folder + +Those comments are only used and parsed consistently in the /lib folder. Outside of this folder the format doesn't follow the convention strictly. Also the comments outside /lib are not used to generate any output. Thus users dont have feedback if the for + ### Unspecified format -In general, the format for writing documentation strings is **not specified**. The only place where it is applied is: *nixpkgs/lib/** +In general, the format for writing documentation strings is **not specified**. + +The *nixdoc*-tool enforces a somewhat consistent format but the actual format was never specified and is not enforced in parts of nixpkgs where nixdoc is currently not applied. -*nixdoc* only applies to places in nixpkgs/lib. But extending the scope of *nixdoc* does not work and thus is not the primary goal. Instead, we should find formal rules for writing *doc-strings*. Tools like *nixdoc* can then implement against this RFC instead of the format relying on nixdoc implementation details. +Extending the scope of *nixdoc* is not the primary goal. Instead, we should find formal rules for writing *doc-comments*. Tools like *nixdoc* can then implement against this RFC instead of the format relying on nixdoc implementation details. ### Only specific placements work @@ -79,7 +80,7 @@ e.g., ### Differentiate from regular comments -The format doesn't allow any distinction between doc-strings or regular comments. +The format doesn't allow any distinction between doc-comments or regular comments. Having a distinction would allow us to @@ -116,22 +117,20 @@ Other tools that work directly with the nix AST and comments: **Proposed Solution:** -Use `##` For doc-string body and markdown headings `# H1` +Use `##` For doc-comment body and markdown headings `# H1` -The content of all doc-strings is markdown. +The content of all doc-comments is markdown. Following the [commonmark-specification](https://spec.commonmark.org/) -## Doc-blocks +## Doc-comments -In general, I thought we do need two kinds of doc-strings: +In general, I thought we do need two kinds of doc-comments: -### A doc-string referencing the subsequent expression +### A doc-comment referencing the subsequent expression -Example: - -```nix -# somefile.nix +`somefile.nix` +```nix { ## ## Documentation for 'mapAttrs' @@ -387,11 +386,9 @@ This section contains examples for the different formats to visualize them and e with `Markdown` Headings -Example: +`somefile.nix` ```nix -# somefile.nix - ## ## ## # Example @@ -408,8 +405,6 @@ Example: With `@{keyword}:` Headings -Example: - ```nix /** @@ -442,7 +437,7 @@ Example: ## Alternative approach - just comments -There is the idea from python that doc-strings are just strings, not even special ones. Comments will be docstrings if they follow specific placement rules. However, we thought this was a bad idea to follow. Such complex placement rules require the users to understand where those places are; with nix syntax, this is slightly more complex than with python. Because we don't have keywords such as `class MyClass():` or `def function():` where placement would be obvious +There is the idea from python that doc-strings are just strings, not even special ones. Strings will be docstrings if they follow specific placement rules. However, we thought this was a bad idea to follow. Such complex placement rules require the users to understand where those places are; with nix syntax, this is slightly more complex than with python. Because we don't have keywords such as `class MyClass():` or `def function():` where placement would be obvious # Unresolved questions [unresolved]: #unresolved-questions @@ -454,11 +449,11 @@ There is the idea from python that doc-strings are just strings, not even specia - How can a tool keep the connection from where a docstring was defined and where the attribute was exposed (lib/default. nix exposes mapAttrs which is defined at lib/attrsets. nix) - There are more complicated things. --> Answer: A Tool might be able to keep track of a percentage of expressions, and sometimes it may be very hard or impossible. For that case, the doc-string can offer a dedicated Keyword to override the scope. +-> Answer: A Tool might be able to keep track of a percentage of expressions, and sometimes it may be very hard or impossible. For that case, the doc-comment can offer a dedicated Keyword to override the scope. e.g. -The following is an idea for a problem that will arise if tools try to track doc-strings' positions and the location in the nixpkgs tree. (Although this problem is not nixpkgs specific) +The following is an idea for a problem that will arise if tools try to track doc-comments' positions and the location in the nixpkgs tree. (Although this problem is not nixpkgs specific) ```nix #| This file is called somewhere that cannot be automatically tracked/is impossible to analyze statically. @@ -483,14 +478,14 @@ Those LSP's should implement the simple "line continuation" feature. (I dont kno ## Nixodc -Nixdoc needs to be changed in order to differentiate between regular comment and doc-blocks. +Nixdoc needs to be changed in order to differentiate between regular comment and doc-comments. There might be an intermediate phase of transition, where the old syntax and features is supported for a while. - When extending nixdoc or writing dedicated parsers, the following persons can assist: [@hsjobeki] ## Documentation generators -Generating documentation from doc-blocks is still a challenge. +Generating documentation from doc-comments is still a challenge. If we'd like the fully automated approach, we definetly need something that can also evaluate nix expressions. (We have such a module in `tvix` which needs to be investigated more here) @@ -506,7 +501,7 @@ As this second approach is much easier I propose this is how we should initially ## Native support in nix -- `NixOS/nix` should implement native support for doc-strings so that our users don't have to rely on nixpkgs or external tools. Those tools can still exist and provide more custom functionality, but documenting your nix expressions should be natively possible. +- `NixOS/nix` should implement native support for doc-comments so that our users don't have to rely on nixpkgs or external tools. Those tools can still exist and provide more custom functionality, but documenting your nix expressions should be natively possible. ## Provide a stable and reliable format @@ -522,7 +517,7 @@ As this second approach is much easier I propose this is how we should initially > People mentioned here might be not yet aware of this rfc. > I'll ping them in the next few days to make sure they are okay with beeing mentioned here. -About doc-strings in general +About doc-comments/doc-strings in general - [@flokli](https://github.com/flokli) - one of the [tvix](https://tvl.fyi/blog/rewriting-nix) authors - [@tazjin](https://github.com/tazjin) - Original Author of `nixdoc`, one of the `tvix` authors @@ -531,7 +526,7 @@ About documenation approaches on independent framworks - [@davHau](https://github.com/davHau) - Author of [dream2nix](https://github.com/nix-community/dream2nix), (And many other nix-frameworks) -About defining weakly typed-interfaces for nix with doc-strings +About defining weakly typed-interfaces for nix with doc-comments - [@roberth](https://github.com/roberth) - nixpkgs Architecture Team - [@aakropotkin](https://github.com/aakropotkin/) - nixpkgs Architecture Team From 498b045b02131340142e6f9752409dbe2009f202 Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Thu, 6 Apr 2023 12:30:51 +0200 Subject: [PATCH 032/110] update --- rfcs/0145-doc-strings.md | 33 ++++++--------------------------- 1 file changed, 6 insertions(+), 27 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 83b5e0d0d..82dcf782a 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -124,7 +124,7 @@ Following the [commonmark-specification](https://spec.commonmark.org/) ## Doc-comments -In general, I thought we do need two kinds of doc-comments: +To explain the concrete usage secnario i'd like to start with a concrete example. ### A doc-comment referencing the subsequent expression @@ -146,46 +146,25 @@ In general, I thought we do need two kinds of doc-comments: } ``` -### A Doc-string referencing the file expression - -Example: Uses `#|` instead of `##` to reference the whole file. It must be at the top of the file. - -```nix -#| -#| -#| # Example -#| -#| -#| -#| # Type -#| -#| - -{...}: -{ - mapAttrs = f: s: <...>; -} -``` - -The following abstract rules describe how to write doc-strings. +The following abstract rules describe how to write doc-comments. > I would be pleased if you comment about whether we should use `## {content}` or `/** {content} */` > I did write this RFC tailored towards `##` but using `/** */` may still be an open discussion. ### Format Rules -- doc-string are all comments. That start with `##` or `#|` e.g. `## {content}` +- doc-comments are all comments. That start with `##` e.g. `## {content}` This RFC is a significant change to the existing documentation convention but we finally need to distinguish between regular comments and doc strings. We found this format to be the most distinguishable. -- Doc-strings always document / relate to an expression. +- Doc-comments always document / relate to an expression. -In detail: `##` relates to the next AST-Node `#|` to the previous one. However technical details for tracking names and aliases is not part of this document. +In detail: `##` relates to the next AST-Node. However technical details for tracking names and aliases is not part of this document. > Vision: Implement a custom evaluator that specializes in tracking references of doc-strings within the nix expression tree. This is however a technical concrete solution that may be build after this rfc is accepted. -- Doc-strings starting with `##` relate to the expression in the following line / or, more precisely, to the next node in the AST. (Implementation Details are not considered yet, maybe future versions need to narrow the scope here) +- Doc-comments starting with `##` relate to the expression in the following line / or, more precisely, to the next node in the AST. (Implementation Details are not considered yet, maybe future versions need to narrow the scope here) > I wont go into details here as this is would already be an implementation specification, but this is how i thought it could technically make sense. > From 6d47a9335767e56b926a5656818c98dc2ea71894 Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Thu, 6 Apr 2023 15:06:13 +0200 Subject: [PATCH 033/110] update --- rfcs/0145-doc-strings.md | 150 ++++++++++++++++----------------------- 1 file changed, 63 insertions(+), 87 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 82dcf782a..208432700 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -13,19 +13,22 @@ related-issues: (will contain links to implementation PRs) Propose a standard format for Doc-comments. - # Motivation [motivation]: #motivation -The following are my proposed goals +The following are the envisioned goals + +- be able to generate documentation from code for any nix expression. +- be able to distinuish between documentation-relevant comments and unrelated comments. +- make doc comments easy to write and read +- be able to parse and render doc comments nicely +- standardize a format for doc comments that can be extended by further RFCs -- Be able to find all documentation related comments -- Be able to generate documentation from code for any nix expression. -- Avoid reading source code unnecssary. Imagine your first time reading nix source code to understand internal functions when beeing totally new to nix. +> Hint: Generating static documentation is a controverse topic in nixpkgs. It was found that is impossible to generate accurate documentation statically. A correct solution would involve evalution of expressions in some way. This already goes deeply into implementation details and is thus not further discussed in this document. ## Current State -We currently utilize a `doc-comment`-like functionality to build a subset of documentation for nix functions. (e.g., nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) +We currently utilize a `doc-comment`-like functionality to build a subset of static documentation for nix functions. (e.g., nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) Many inconsistently written comments document specific parts of nixpkgs and other nix-frameworks (see [references-to-this](#references-to-the-problems-above)). We use some of them to generate documentation automatically. (e.g., nixpkgs/lib via [nixdoc](https://github.com/nix-community/nixdoc) ) @@ -58,7 +61,7 @@ Here is an example of the format understood by *nixdoc*: ### Inconsistend usage outside /lib folder -Those comments are only used and parsed consistently in the /lib folder. Outside of this folder the format doesn't follow the convention strictly. Also the comments outside /lib are not used to generate any output. Thus users dont have feedback if the for +Those comments are only used and parsed consistently in the /lib folder. Outside of this folder the format doesn't follow the convention strictly. Also the comments outside /lib are not used to generate any output. ### Unspecified format @@ -70,7 +73,7 @@ Extending the scope of *nixdoc* is not the primary goal. Instead, we should find ### Only specific placements work -The placement of those comments requires precisely commenting at the attribute set containing the function declarations, which is not usable for general-purpose documentation strings. +The placement of those comments requires precisely commenting at the attribute set containing the function declarations, which is not usable for general-purpose documentation strings. e.g., @@ -108,7 +111,7 @@ Having a distinction would allow us to Other tools that work directly with the nix AST and comments: -- [noogle](https://noogle.dev) - Nix API search engine. It allows you to search functions and other expressions. +- [noogle](https://noogle.dev) - Nix API search engine. It allows to search functions and other expressions. - [nix-doc](https://github.com/lf-/nix-doc) - A Nix developer tool leveraging the rnix Nix parser for intelligent documentation search and tags generation - [manix](https://github.com/mlvzk/manix) - A fast CLI documentation searcher for nix. @@ -117,14 +120,14 @@ Other tools that work directly with the nix AST and comments: **Proposed Solution:** -Use `##` For doc-comment body and markdown headings `# H1` +Use `##` For doc-comment body and markdown within -The content of all doc-comments is markdown. +The content of all doc-comments is rednered using markdown. Following the [commonmark-specification](https://spec.commonmark.org/) ## Doc-comments -To explain the concrete usage secnario i'd like to start with a concrete example. +The following exampple demonstrates the concrete usage scenario. ### A doc-comment referencing the subsequent expression @@ -146,112 +149,85 @@ To explain the concrete usage secnario i'd like to start with a concrete example } ``` -The following abstract rules describe how to write doc-comments. - -> I would be pleased if you comment about whether we should use `## {content}` or `/** {content} */` -> I did write this RFC tailored towards `##` but using `/** */` may still be an open discussion. +## Format Rules -### Format Rules +The following abstract rules describe how to write doc-comments. -- doc-comments are all comments. That start with `##` e.g. `## {content}` +### doc-comments are all comments. That start with `##` e.g. `## {content}` This RFC is a significant change to the existing documentation convention -but we finally need to distinguish between regular comments and doc strings. We found this format to be the most distinguishable. +but allows to distinguish between regular comments and doc comments. This is important because arbitrary code comments should not end up in generated documentation. -- Doc-comments always document / relate to an expression. +### Doc-comments always document / relate to an expression. In detail: `##` relates to the next AST-Node. However technical details for tracking names and aliases is not part of this document. -> Vision: Implement a custom evaluator that specializes in tracking references of doc-strings within the nix expression tree. This is however a technical concrete solution that may be build after this rfc is accepted. - -- Doc-comments starting with `##` relate to the expression in the following line / or, more precisely, to the next node in the AST. (Implementation Details are not considered yet, maybe future versions need to narrow the scope here) +> Vision: Implement a custom evaluator that specializes in tracking references of doc-comments within the nix expression tree. This is however a technical concrete solution that may be build after this rfc is accepted. -> I wont go into details here as this is would already be an implementation specification, but this is how i thought it could technically make sense. -> -> E.g. a "documentation generator": -> The referenced expression might be an IDENT_NODE or an expression that can be assigned to an IDENT_NODE. -> The doc-string may then be yielded from the IDENT_NODE, or from the expression that was assigned to the NODE. (Precendence is -1, which means a doc-string is always the last assignment) -> ```nix -> ## Refences the whole thing -> ## (foo bar baz) -> foo bar baz -> ``` -> Expressions would thus need a meta-wrapper that hold the necessary doc-string and preserves it during the evaluation. While rendering out the whole nix expression (recursion need to be considered e.g. in Derivation, etc.!) All references are preserved and can then be mapped into a list for each identifier in the tree. -> e.g. `foo.bar.baz` holds a refernce to the docstring that was asigned to the expression in `baz.nix` -> -> BUT this is a very concrete documentation generator, that makes eventually sense to be built after this RFC. +### Doc-comments start with `##` -- Doc-strings that are at the top of a file and that start with `#|` describe the expression exported from the whole file. (Previous node in AST) +Doc-comments start with `##` (leading whitespaces are allowed). -In comparison, rustdoc uses `//!`. But using `#!` is considered a bad idea, as it can be confused with identical bash shebangs `#!`. - -The `|` (pipe) is also available in symbols used for the nix grammar. - -> This is still being determined. If you have any ideas, let us know in the comments. - -Example of a comment referring to the whole file: - -```nix -#| -#| -#| # Example -#| -#| # Type -#| The Type of the expression returned by the file -{ - id = x: x -} -``` +Doc-comments relate to the expression in the following line or, more precisely, to the next node in the AST. -- The docstring is continued in the following line if it also starts with `##` / `#|`. Leading whitespace is allowed. +The docstring is continued in the following line if it also starts with `##`. +Leading whitespace is allowed. Example: docstring continuation ```nix -## Doc-string A +## doc-comment A ## .... ## This block has no expression in the following line. ## Therefore, it doesn't have any effect ## .... -## Doc-string A +## doc-comment A -## Doc-string B +## doc-comment B ## -- This block documents the purpose of '1' -## Doc-string B +## doc-comment B 1 ``` -- The content of a doc-string is Markdown. +It is common using the same indentation as the referenced expression. + +```nix + +``` + +### The content of a doc-comment is Markdown. + +The content of a doc-comment is parsed using Markdown following the commonmark specification. Thus it allows for intuitive usage without knowledge of complex syntax rules. + +Top level H1 headings, starting with a single `#`, indicate sections. Some of which might be spcified in future rfcs. + +Common Sections: -It allows for intuitive usage without knowledge of complex syntax rules. +- `# Examples` +- `# Type` -- predefined heading `keywords` may start a section. -- Content before the first [optional] section is called `description`. +Sections that might be specified in future rfcs: -This allows for quick writing without the need to use sections. +- `# Type` +- `# Arguments` +- `# Meta` -- Headings H1 are reserved markdown headings. Which are specified in [this list](#keywords). Users are allowed to only use H2 (or higher) headings for their free use. +Until they are actually specified. -H1 headings start a section to keep it extendable in the future. Users are not allowed to choose them freely, so we keep track of all allowed H1 headings. +Future RFCs may specify sub-headings of `# Meta`. Usage is reserved. -This may also be checked from the doc-tool that may evolve from this RFC (e.g. future versions of nixdoc) +Possible sub-headings: -- Every [optional] section started by an H1 heading is continued until the next heading starts. To the very end of the comment, otherwise. -- Every section may define its own rules. They must be compatible with the formal requirements of doc-strings (this RFC) that can override formal rules locally. (e.g., disable Markdown, use custom syntax, etc.) -- Only the H1-sections (`Keywords`) described in [this list](#keywords) are valid. - -- In case of [future] extensions, every new section `Keyword` must first be added to this RFC. -- If sections follow complex logic, it is embraced to specify that logic in a separate sub-RFC. -- Usage of the described sections is OPTIONAL. -- more tbd. +- `## AttrPath` +- `## ` ## Keywords [keywords]: #keywords The following keywords start new markdown sections -> I wanted to keep the list of initial keywords short. So by the time this RFC focuses on the formal aspects of doc-strings first. More keywords and features for them may be added later on. +> I wanted to keep the list of initial keywords short. So by the time this RFC focuses on the formal aspects of doc-comments first. More keywords and features for them may be added later on. | Keyword | Description | Note | | --- | --- | --- | @@ -268,16 +244,16 @@ First, there are no actual block specifiers within nix or nixpkgs. The existing The sequence `Example:` has some drawbacks when it comes to syntax: -1. It is possible that this sequence occurs in a natural text without the intention to start a new doc-string section. +1. It is possible that this sequence occurs in a natural text without the intention to start a new doc-comment section. 2. It doesn't visually stand out. 3. It is terrible that the line needs to start with `Example:` to be valid syntax. However, it is a good practice while writing comments; it should be optional. 4. It neither follows the `@param` (c/c++/java,...) convention nor the markdown headings convention (rust); instead is nixdoc-home-cooked. ## Interactions -Doc-strings can be attached to AST nodes without affecting the actual compile-, evaluation- or build-time because they are just comments. Specialized tools can handle those comments and create static documentation from them. Also, integration with LSP is possible. (See [@flokli's nix lsp-whishlist](https://hackmd.io/@geyA7YL_RyiWJO6d5TbC-g/Sy6lVrgW3) for inspirations) +doc-comments can be attached to AST nodes without affecting the actual compile-, evaluation- or build-time because they are just comments. Specialized tools can handle those comments and create static documentation from them. Also, integration with LSP is possible. (See [@flokli's nix lsp-whishlist](https://hackmd.io/@geyA7YL_RyiWJO6d5TbC-g/Sy6lVrgW3) for inspirations) -Following this RFC means refactoring for existing comments, but it also means that **we can finally use all comments (automated!) that were intended to be doc-strings** +Following this RFC means refactoring for existing comments, but it also means that **we can finally use all comments (automated!) that were intended to be doc-comments** # Drawbacks [drawbacks]: #drawbacks @@ -298,7 +274,7 @@ While designing this RFC multiple alternative formats where considered. They can In general, we needed the following: -1. General format for doc-strings. +1. General format for doc-comments. 2. Format for headings and the allowed content. > It would be nice if this could be close to the markdown format. @@ -322,7 +298,7 @@ In general, we needed the following: **Proposed format:** -Use `##` to start a doc-string. This allows clear visual seperation from regular comments. +Use `##` to start a doc-comment. This allows clear visual seperation from regular comments. And provides a good compatibiliy with the strived markdown content. ### Refactoring note: @@ -416,7 +392,7 @@ With `@{keyword}:` Headings ## Alternative approach - just comments -There is the idea from python that doc-strings are just strings, not even special ones. Strings will be docstrings if they follow specific placement rules. However, we thought this was a bad idea to follow. Such complex placement rules require the users to understand where those places are; with nix syntax, this is slightly more complex than with python. Because we don't have keywords such as `class MyClass():` or `def function():` where placement would be obvious +There is the idea from python that doc-comments are just strings, not even special ones. Strings will be docstrings if they follow specific placement rules. However, we thought this was a bad idea to follow. Such complex placement rules require the users to understand where those places are; with nix syntax, this is slightly more complex than with python. Because we don't have keywords such as `class MyClass():` or `def function():` where placement would be obvious # Unresolved questions [unresolved]: #unresolved-questions @@ -480,7 +456,7 @@ As this second approach is much easier I propose this is how we should initially ## Native support in nix -- `NixOS/nix` should implement native support for doc-comments so that our users don't have to rely on nixpkgs or external tools. Those tools can still exist and provide more custom functionality, but documenting your nix expressions should be natively possible. +- `NixOS/nix` should implement native support for doc-comments so that our users don't have to rely on nixpkgs or external tools. Those tools can still exist and provide more custom functionality, but documenting nix expressions should be natively possible. ## Provide a stable and reliable format @@ -496,7 +472,7 @@ As this second approach is much easier I propose this is how we should initially > People mentioned here might be not yet aware of this rfc. > I'll ping them in the next few days to make sure they are okay with beeing mentioned here. -About doc-comments/doc-strings in general +About doc-comments/doc-comments in general - [@flokli](https://github.com/flokli) - one of the [tvix](https://tvl.fyi/blog/rewriting-nix) authors - [@tazjin](https://github.com/tazjin) - Original Author of `nixdoc`, one of the `tvix` authors From 14dca53f5cde23ed3dadff2df88a989ae8627419 Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Thu, 6 Apr 2023 15:14:37 +0200 Subject: [PATCH 034/110] update --- rfcs/0145-doc-strings.md | 70 +++++++++++++++++----------------------- 1 file changed, 30 insertions(+), 40 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 208432700..03b25dcac 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -19,12 +19,12 @@ Propose a standard format for Doc-comments. The following are the envisioned goals - be able to generate documentation from code for any nix expression. -- be able to distinuish between documentation-relevant comments and unrelated comments. +- be able to distinguish between documentation-relevant comments and unrelated comments. - make doc comments easy to write and read - be able to parse and render doc comments nicely - standardize a format for doc comments that can be extended by further RFCs -> Hint: Generating static documentation is a controverse topic in nixpkgs. It was found that is impossible to generate accurate documentation statically. A correct solution would involve evalution of expressions in some way. This already goes deeply into implementation details and is thus not further discussed in this document. +> Hint: Generating static documentation is controvert topic in nixpkgs. It was found that is impossible to generate accurate documentation statically. A correct solution would involve evaluation of expressions in some way. This already goes deeply into implementation details and is thus not further discussed in this document. ## Current State @@ -59,9 +59,9 @@ Here is an example of the format understood by *nixdoc*: ## Current problems -### Inconsistend usage outside /lib folder +### Inconsistent usage outside /lib folder -Those comments are only used and parsed consistently in the /lib folder. Outside of this folder the format doesn't follow the convention strictly. Also the comments outside /lib are not used to generate any output. +Those comments are only used and parsed consistently in the /lib folder. Outside of this folder the format doesn't follow the convention strictly. Also the comments outside /lib are not used to generate any output. ### Unspecified format @@ -69,13 +69,13 @@ In general, the format for writing documentation strings is **not specified**. The *nixdoc*-tool enforces a somewhat consistent format but the actual format was never specified and is not enforced in parts of nixpkgs where nixdoc is currently not applied. -Extending the scope of *nixdoc* is not the primary goal. Instead, we should find formal rules for writing *doc-comments*. Tools like *nixdoc* can then implement against this RFC instead of the format relying on nixdoc implementation details. +Extending the scope of *nixdoc* is not the primary goal. Instead, we should find formal rules for writing *doc-comments*. Tools like *nixdoc* can then implement against this RFC instead of the format relying on nixdoc implementation details. ### Only specific placements work The placement of those comments requires precisely commenting at the attribute set containing the function declarations, which is not usable for general-purpose documentation strings. -e.g., +e.g., - file that directly exports the lib-function without wrapping it in an attribute set. - file that exports a constant expression @@ -113,16 +113,16 @@ Other tools that work directly with the nix AST and comments: - [noogle](https://noogle.dev) - Nix API search engine. It allows to search functions and other expressions. - [nix-doc](https://github.com/lf-/nix-doc) - A Nix developer tool leveraging the rnix Nix parser for intelligent documentation search and tags generation -- [manix](https://github.com/mlvzk/manix) - A fast CLI documentation searcher for nix. +- [manix](https://github.com/mlvzk/manix) - A fast CLI documentation searcher for nix. # Detailed design [design]: #detailed-design -**Proposed Solution:** +**Proposed Solution:** Use `##` For doc-comment body and markdown within -The content of all doc-comments is rednered using markdown. +The content of all doc-comments is rendered using markdown. Following the [commonmark-specification](https://spec.commonmark.org/) ## Doc-comments @@ -158,9 +158,9 @@ The following abstract rules describe how to write doc-comments. This RFC is a significant change to the existing documentation convention but allows to distinguish between regular comments and doc comments. This is important because arbitrary code comments should not end up in generated documentation. -### Doc-comments always document / relate to an expression. +### Doc-comments always document / relate to an expression -In detail: `##` relates to the next AST-Node. However technical details for tracking names and aliases is not part of this document. +In detail: `##` relates to the next AST-Node. However technical details for tracking names and aliases is not part of this document. > Vision: Implement a custom evaluator that specializes in tracking references of doc-comments within the nix expression tree. This is however a technical concrete solution that may be build after this rfc is accepted. @@ -196,11 +196,11 @@ It is common using the same indentation as the referenced expression. ``` -### The content of a doc-comment is Markdown. +### The content of a doc-comment is Markdown The content of a doc-comment is parsed using Markdown following the commonmark specification. Thus it allows for intuitive usage without knowledge of complex syntax rules. -Top level H1 headings, starting with a single `#`, indicate sections. Some of which might be spcified in future rfcs. +Top level H1 headings, starting with a single `#`, indicate sections. Some of which might be specified in future rfcs. Common Sections: @@ -215,12 +215,7 @@ Sections that might be specified in future rfcs: Until they are actually specified. -Future RFCs may specify sub-headings of `# Meta`. Usage is reserved. - -Possible sub-headings: - -- `## AttrPath` -- `## ` +Future RFCs may specify sub-headings of `# Meta`. Its usage is reserved. ## Keywords [keywords]: #keywords @@ -266,7 +261,6 @@ This could mostly be automated. (e.g via codemod) Also, this affects only the `lib` folder and a few other places that are currently used to build the documentation. - # Alternatives [alternatives]: #alternatives @@ -278,7 +272,7 @@ In general, we needed the following: 2. Format for headings and the allowed content. > It would be nice if this could be close to the markdown format. -> Markdown is the most straightforward and most accepted format +> Markdown is the most straightforward and most accepted format > for writing and rendering documentation. ## General Formats @@ -298,16 +292,16 @@ In general, we needed the following: **Proposed format:** -Use `##` to start a doc-comment. This allows clear visual seperation from regular comments. -And provides a good compatibiliy with the strived markdown content. +Use `##` to start a doc-comment. This allows clear visual separation from regular comments. +And provides a good compatibility with the strived markdown content. -### Refactoring note: +### Refactoring note nixpkgs comments: -- `##` ~4k usages (most of them are used for visual seperation e.g. `###########`) -- `#` ~20k usages -- `/*` ~6k usages +- `##` ~4k usages (most of them are used for visual separation e.g. `###########`) +- `#` ~20k usages +- `/*` ~6k usages - `/**` 160 usages (most of them are completely empty ?) ## General Headings @@ -322,13 +316,9 @@ nixpkgs comments: **Proposed headings:** -Use markdown headings `# `. This allows best compatibility with the aleady specified markdown/commonmark format. Allowing for easy and intuitive usage for comments. - -## Why we should do this +Use markdown headings `# `. This allows best compatibility with the already specified markdown/commonmark format. Allowing for easy and intuitive usage for comments. -- Find all comments that are part of the documentation -- Render them using markdown -- Connect the documentation to the exact places in the nix code. +## Consequences to not implementing this - By not implementing this feature, nix loses the ability for tool-generated documentation. - Documentation will not defined by nixdoc, instead the community-implementation solution to the standard. @@ -401,7 +391,7 @@ There is the idea from python that doc-comments are just strings, not even speci - Will `nix` itself implement native support like in rust -> `cargo doc` -- How can a tool keep the connection from where a docstring was defined and where the attribute was exposed (lib/default. nix exposes mapAttrs which is defined at lib/attrsets. nix) +- How can a tool keep the connection from where a docstring was defined and where the attribute was exposed (lib/default. nix exposes mapAttrs which is defined at lib/attrsets.nix) - There are more complicated things. -> Answer: A Tool might be able to keep track of a percentage of expressions, and sometimes it may be very hard or impossible. For that case, the doc-comment can offer a dedicated Keyword to override the scope. @@ -428,8 +418,8 @@ The following is an idea for a problem that will arise if tools try to track doc ## Editor support When starting hitting {enter} inside a doc-block the new line, should be automatically prefixed with `##` or `#|` accordingly. -This is done in rust similarily. Nix already offers a bunch of LSP's e.g. [nil](https://github.com/oxalica/nil), [rnix-lsp](https://github.com/nix-community/rnix-lsp) are the most common ones. -Those LSP's should implement the simple "line continuation" feature. (I dont know the exact name here) +This is done in rust similarly. Nix already offers a bunch of LSP's e.g. [nil](https://github.com/oxalica/nil), [rnix-lsp](https://github.com/nix-community/rnix-lsp) are the most common ones. +Those LSP's should implement the simple "line continuation" feature. (I don't know the exact name here) ## Nixodc @@ -441,7 +431,7 @@ There might be an intermediate phase of transition, where the old syntax and fea ## Documentation generators Generating documentation from doc-comments is still a challenge. -If we'd like the fully automated approach, we definetly need something that can also evaluate nix expressions. +If we'd like the fully automated approach, we definitely need something that can also evaluate nix expressions. (We have such a module in `tvix` which needs to be investigated more here) Alternatively we can use the future nixdoc together with a static `map.json` that contains the scope for every discovery file/path in nixpkgs. @@ -467,17 +457,17 @@ As this second approach is much easier I propose this is how we should initially - [Rustdoc](https://doc.rust-lang.org/rustdoc/how-to-write-documentation.html) - [Nixdoc](https://github.com/nix-community/nixdoc) -## People that I discussed with +## People that I discussed with > People mentioned here might be not yet aware of this rfc. -> I'll ping them in the next few days to make sure they are okay with beeing mentioned here. +> I'll ping them in the next few days to make sure they are okay with being mentioned here. About doc-comments/doc-comments in general - [@flokli](https://github.com/flokli) - one of the [tvix](https://tvl.fyi/blog/rewriting-nix) authors - [@tazjin](https://github.com/tazjin) - Original Author of `nixdoc`, one of the `tvix` authors -About documenation approaches on independent framworks +About documentation approaches on independent frameworks - [@davHau](https://github.com/davHau) - Author of [dream2nix](https://github.com/nix-community/dream2nix), (And many other nix-frameworks) From fe8ab29bd195372865e6b205406f4ea3aaebcbb6 Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Thu, 6 Apr 2023 15:31:24 +0200 Subject: [PATCH 035/110] work feedback into it --- rfcs/0145-doc-strings.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 03b25dcac..d9a32f069 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -24,7 +24,10 @@ The following are the envisioned goals - be able to parse and render doc comments nicely - standardize a format for doc comments that can be extended by further RFCs -> Hint: Generating static documentation is controvert topic in nixpkgs. It was found that is impossible to generate accurate documentation statically. A correct solution would involve evaluation of expressions in some way. This already goes deeply into implementation details and is thus not further discussed in this document. +This RFC is a significant change to the existing documentation convention +but allows to distinguish between regular comments and doc comments. This is important because arbitrary code comments should not end up in generated documentation. + +> Hint: Generating static documentation is controvert topic in nixpkgs. It was found that is impossible to generate accurate documentation statically. A correct solution would involve evaluation of expressions in some way. This already goes deeply into implementation details and is thus not further discussed in this document. Although we envision solutions to solve this issue. ## Current State @@ -155,8 +158,11 @@ The following abstract rules describe how to write doc-comments. ### doc-comments are all comments. That start with `##` e.g. `## {content}` -This RFC is a significant change to the existing documentation convention -but allows to distinguish between regular comments and doc comments. This is important because arbitrary code comments should not end up in generated documentation. +```nix +## Documentation +## follows simple rules +## and can span multiple lines +``` ### Doc-comments always document / relate to an expression @@ -233,8 +239,6 @@ The following keywords start new markdown sections First, there are no actual block specifiers within nix or nixpkgs. The existing blocks heavily depend on a tool called `nixdoc` and not vice versa. --> See [github:nix-community/nixdoc](https://github.com/nix-community/nixdoc) - > `nixdoc` MUST be changed to support this RFC. (See [Future work](#future-work)) The sequence `Example:` has some drawbacks when it comes to syntax: @@ -284,10 +288,10 @@ In general, we needed the following: | Needs Termination | No | Yes | Yes | | Indentation | Clear | Poor | Poor | | Needs vertical space | No | Yes | Yes | -| Visual distinction from comments | High | Low | Intermediate | +| Visual distinction from comments | High | Low | Medium | | Needs Autocompletion (Language Server) to continue the next line. | Yes | No | Yes | | Punctuation Variations / Amount of different special characters | Less | More | More | -| Markdown compatibility (also depends on indentation clarity) | Good, but visual conflicts with headings` #` | Poor | Intermediate | +| Markdown compatibility (also depends on indentation clarity) | Good, but visual conflicts with headings` #` | Poor | Medium | | breaks when interrupted with newlines | Yes | No | ? | **Proposed format:** From b581941393e59f84cb7bbcdb697442a34901b94d Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Thu, 6 Apr 2023 15:54:32 +0200 Subject: [PATCH 036/110] point to type checking --- rfcs/0145-doc-strings.md | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index d9a32f069..b3eab2d2e 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -215,7 +215,7 @@ Common Sections: Sections that might be specified in future rfcs: -- `# Type` +- `# Type` - `# Arguments` - `# Meta` @@ -226,14 +226,16 @@ Future RFCs may specify sub-headings of `# Meta`. Its usage is reserved. ## Keywords [keywords]: #keywords -The following keywords start new markdown sections +The following keywords start reserved markdown sections > I wanted to keep the list of initial keywords short. So by the time this RFC focuses on the formal aspects of doc-comments first. More keywords and features for them may be added later on. | Keyword | Description | Note | | --- | --- | --- | -| `Example` | Starts the Example-block. Often contains comprehensive code examples | | -| `Type` | Start the Type-block; it is any free text | Syntax may eventually be specified in the future. [preview](https://typednix.dev). | +| `# Examples` | Starts the Example-block. Often contains comprehensive code examples | | +| `# Type` | Start the Type-block; Just any free text; but we recommend following the existing convention of the current `Type:` field | Syntax may eventually be specified in the future. | +| `# Meta` | Under this section future rfcs may specify their sub-sections. | Sub-sections within meta will avoid namespace collisions in future RFCs | +| `# Arguments` | Needed for describing the functions arguments. Is just any free text until specified | | ## Why change the existing section specifiers? @@ -461,6 +463,15 @@ As this second approach is much easier I propose this is how we should initially - [Rustdoc](https://doc.rust-lang.org/rustdoc/how-to-write-documentation.html) - [Nixdoc](https://github.com/nix-community/nixdoc) +## Further + +We envision gradual type checking for nix. + +A weak source of type constraint could be the `# Type` field in doc-comments until nix may introduce its own native type system. +Very concrete doc-typing-syntax may allow gradual type checking. + +- see a [preview](https://typednix.dev) of an eventual future doc-type-syntax. + ## People that I discussed with > People mentioned here might be not yet aware of this rfc. From 1f5bf5ea4c7d9b7cf616f7cdaffc5240d4fa8fa4 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 6 Apr 2023 18:31:34 +0200 Subject: [PATCH 037/110] Update rfcs/0145-doc-strings.md Co-authored-by: Anderson Torres --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index b3eab2d2e..4def09e43 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -326,7 +326,7 @@ Use markdown headings `# `. This allows best compatibility with the alr ## Consequences to not implementing this -- By not implementing this feature, nix loses the ability for tool-generated documentation. +- By not implementing this feature, nix gains no ability for tool-generated documentation. - Documentation will not defined by nixdoc, instead the community-implementation solution to the standard. ## Examples From c8e85f3aadcbf96e67131c12ae36452346a3ec24 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Fri, 7 Apr 2023 15:41:22 +0200 Subject: [PATCH 038/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 4def09e43..fa6c1baf3 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -327,7 +327,8 @@ Use markdown headings `# `. This allows best compatibility with the alr ## Consequences to not implementing this - By not implementing this feature, nix gains no ability for tool-generated documentation. -- Documentation will not defined by nixdoc, instead the community-implementation solution to the standard. +- Documentation will be defined by nixdoc, not by the nix community itself. +- Many existing comments written for documentation will remain un-discoverable. ## Examples @@ -407,10 +408,14 @@ e.g. The following is an idea for a problem that will arise if tools try to track doc-comments' positions and the location in the nixpkgs tree. (Although this problem is not nixpkgs specific) ```nix -#| This file is called somewhere that cannot be automatically tracked/is impossible to analyze statically. -#| The 'TreePath' override can be used by the docstring author to set a fixed path in the nixpkgs expression. -#| (This behavior will not be specified in this RFC) -#| @TreePath: pkgs.stdenv +## This file is called somewhere that cannot be automatically tracked/is impossible to analyze statically. +## The 'TreePath' override can be used by the docstring author to set fixed paths in the nixpkgs expression. +## This tells the doc-tool that the expression is available in two aliases: `pkgs.stdenv` and `lib.sth` +## (This behavior will not be specified in this RFC) +## # Meta +## ## TreePath +## - pkgs.stdenv +## - lib.sth {...}: { From 5da22b43a3af6b580ab988c051b741f25241185a Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sat, 8 Apr 2023 19:15:49 +0200 Subject: [PATCH 039/110] Add roberth's PR from 2017 --- rfcs/0145-doc-strings.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index fa6c1baf3..2e4560290 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -470,6 +470,10 @@ As this second approach is much easier I propose this is how we should initially ## Further +There is an actual but rather old PR that uses just comments to show documentation in the nix repl for functions. + +- https://github.com/NixOS/nix/pull/1652 + We envision gradual type checking for nix. A weak source of type constraint could be the `# Type` field in doc-comments until nix may introduce its own native type system. From 434f8b4b9de78b49b9c61586653f8dc88e35ad0c Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sat, 8 Apr 2023 20:27:56 +0200 Subject: [PATCH 040/110] Update rfcs/0145-doc-strings.md Co-authored-by: gilice <104317939+gilice@users.noreply.github.com> --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 2e4560290..6be0c3c46 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -36,7 +36,7 @@ Many inconsistently written comments document specific parts of nixpkgs and othe We use some of them to generate documentation automatically. (e.g., nixpkgs/lib via [nixdoc](https://github.com/nix-community/nixdoc) ) -This solution requires a lot of handworks; more specifically, *nixdoc* is a custom tool that works only for that purpose. +This solution requires a lot of manual work; more specifically, *nixdoc* is a custom tool that works only for that purpose. Here is an example of the format understood by *nixdoc*: From 1105ea3c9a9c1678963a48de1cfa939602abac5d Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sat, 8 Apr 2023 20:28:08 +0200 Subject: [PATCH 041/110] Update rfcs/0145-doc-strings.md Co-authored-by: gilice <104317939+gilice@users.noreply.github.com> --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 6be0c3c46..48868124f 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -206,7 +206,7 @@ It is common using the same indentation as the referenced expression. The content of a doc-comment is parsed using Markdown following the commonmark specification. Thus it allows for intuitive usage without knowledge of complex syntax rules. -Top level H1 headings, starting with a single `#`, indicate sections. Some of which might be specified in future rfcs. +Top level H1 headings, starting with a single `#`, indicate sections. Sections might be specified in future rfcs. Common Sections: From cb7f9d6f3a03a72547d5131400e066fd1cf3c5dd Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sat, 8 Apr 2023 20:57:08 +0200 Subject: [PATCH 042/110] Update rfcs/0145-doc-strings.md Co-authored-by: gilice <104317939+gilice@users.noreply.github.com> --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 48868124f..18b5fba48 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -123,7 +123,7 @@ Other tools that work directly with the nix AST and comments: **Proposed Solution:** -Use `##` For doc-comment body and markdown within +Each line of a doc-comment starts with `##`. The content of all doc-comments is rendered using markdown. Following the [commonmark-specification](https://spec.commonmark.org/) From e5397ee725003f16ca551ea245c03d5126de1022 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sat, 8 Apr 2023 21:14:26 +0200 Subject: [PATCH 043/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 18b5fba48..6dab66078 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -199,7 +199,8 @@ Example: docstring continuation It is common using the same indentation as the referenced expression. ```nix - + ## The function foo makes us happy + foo = arg: #... ``` ### The content of a doc-comment is Markdown @@ -428,9 +429,11 @@ The following is an idea for a problem that will arise if tools try to track doc ## Editor support -When starting hitting {enter} inside a doc-block the new line, should be automatically prefixed with `##` or `#|` accordingly. +When starting hitting {enter} inside a doc-block the new line, should be automatically prefixed with `##` accordingly. This is done in rust similarly. Nix already offers a bunch of LSP's e.g. [nil](https://github.com/oxalica/nil), [rnix-lsp](https://github.com/nix-community/rnix-lsp) are the most common ones. -Those LSP's should implement the simple "line continuation" feature. (I don't know the exact name here) + +- Those LSP's should implement the simple "line continuation" feature. (I don't know the exact name here) +- Implement displaying the related documentation when hovering over an expression. (lspAction/hover) ## Nixodc From 403b8cdf9463fbf52dfaa5ffc436993920dc898e Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 9 Apr 2023 12:15:33 +0200 Subject: [PATCH 044/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 39 ++++++++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 15 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 6dab66078..f00c50f88 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -170,12 +170,10 @@ In detail: `##` relates to the next AST-Node. However technical details for trac > Vision: Implement a custom evaluator that specializes in tracking references of doc-comments within the nix expression tree. This is however a technical concrete solution that may be build after this rfc is accepted. -### Doc-comments start with `##` +### Doc-comments continue with `##` Doc-comments start with `##` (leading whitespaces are allowed). -Doc-comments relate to the expression in the following line or, more precisely, to the next node in the AST. - The docstring is continued in the following line if it also starts with `##`. Leading whitespace is allowed. @@ -196,7 +194,7 @@ Example: docstring continuation 1 ``` -It is common using the same indentation as the referenced expression. +> Hint: It is common to use the same indentation as the referenced expression. ```nix ## The function foo makes us happy @@ -205,7 +203,7 @@ It is common using the same indentation as the referenced expression. ### The content of a doc-comment is Markdown -The content of a doc-comment is parsed using Markdown following the commonmark specification. Thus it allows for intuitive usage without knowledge of complex syntax rules. +The content of a doc-comment is parsed using Markdown following the [commonmark specification](https://commonmark.org/). Thus it allows for intuitive usage without knowledge of complex syntax rules. Top level H1 headings, starting with a single `#`, indicate sections. Sections might be specified in future rfcs. @@ -229,7 +227,7 @@ Future RFCs may specify sub-headings of `# Meta`. Its usage is reserved. The following keywords start reserved markdown sections -> I wanted to keep the list of initial keywords short. So by the time this RFC focuses on the formal aspects of doc-comments first. More keywords and features for them may be added later on. +> We wanted to keep the list of initial keywords short. So by the time this RFC focuses on the formal aspects of doc-comments first. More keywords and features for them may be added later on. | Keyword | Description | Note | | --- | --- | --- | @@ -331,13 +329,17 @@ Use markdown headings `# `. This allows best compatibility with the alr - Documentation will be defined by nixdoc, not by the nix community itself. - Many existing comments written for documentation will remain un-discoverable. +## Enforcing lsp support is not great + +Single-line comments `##` require using a Language Server (LSP) to make developers work productively. It is desirable to have a doc-comment format that works nicely without an LSP. To achieve this with a variant of the multiline comment `/**{content}*/` as the format for doc-comments to be the respective subset of multiline comments + ## Examples This section contains examples for the different formats to visualize them and emphasize the previously discussed characteristics. ### `##` inspired from rust's `///` -with `Markdown` Headings +Example of the proposed `##` format; with `Markdown` Headings. `somefile.nix` @@ -356,19 +358,24 @@ with `Markdown` Headings ### `/** */` inspired by the current multiline strings -With `@{keyword}:` Headings +Example of an alternative multiline`/** */` format; with `Markdown` Headings. + +TODO: Work out the details for multiline doc comments together with documentation-team. This section might be removed and included into the "detailed design" section. ```nix /** - + mapAttrs is a function in lib - @Example: + # Example - + ``` + comprehensive example + shows how to use this + ``` - @Type: + # Type - + foo -> bar */ mapAttrs = f: s: #... ``` @@ -376,6 +383,8 @@ With `@{keyword}:` Headings ## Javadoc style +Example of the discarded `javadoc` format. + ```java /** * A short description @@ -450,7 +459,7 @@ If we'd like the fully automated approach, we definitely need something that can Alternatively we can use the future nixdoc together with a static `map.json` that contains the scope for every discovery file/path in nixpkgs. -As this second approach is much easier I propose this is how we should initially start to extend the scope. +As this second approach is much easier; We propose this is how we should initially start to extend the scope. ## More specialized section headings @@ -484,7 +493,7 @@ Very concrete doc-typing-syntax may allow gradual type checking. - see a [preview](https://typednix.dev) of an eventual future doc-type-syntax. -## People that I discussed with +## People discussed with > People mentioned here might be not yet aware of this rfc. > I'll ping them in the next few days to make sure they are okay with being mentioned here. From 40a51a16e096d836644ee116981349c086fd58b7 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Wed, 12 Apr 2023 15:48:24 +0200 Subject: [PATCH 045/110] add: point to the concerns --- rfcs/0145-doc-strings.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index f00c50f88..2cffc3b29 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -3,7 +3,7 @@ feature: doc-comment-standard start-date: 2023-03-27 author: hsjobeki co-authors: (find a buddy later to help out with the RFC) -shepherd-team: (names, to be nominated and accepted by RFC steering committee) +shepherd-team: @DavHau, ? (If you are interested please PM the author on matrix or comment) shepherd-leader: (name to be appointed by RFC steering committee) related-issues: (will contain links to implementation PRs) --- @@ -13,6 +13,13 @@ related-issues: (will contain links to implementation PRs) Propose a standard format for Doc-comments. +This RFC includes two separate concerns: + +- Outer format rules to allow distinction between regular comments and doc-comments +- Inner format rules that describe the required format of the content. + +However, both concerns relate closely to each other; We thought it made sense and reduced bureaucracy to address that in a single RFC. + # Motivation [motivation]: #motivation From 254863e4b8473bbaad33ee2f048037f4509c0451 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Wed, 12 Apr 2023 15:56:50 +0200 Subject: [PATCH 046/110] add: motivation multiline comments --- rfcs/0145-doc-strings.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 2cffc3b29..a6b7deffb 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -88,8 +88,16 @@ The placement of those comments requires precisely commenting at the attribute s e.g., - file that directly exports the lib-function without wrapping it in an attribute set. -- file that exports a constant expression -- files outside of lib cannot be rendered due to missing conventions +- file that exports a constant expression. +- files outside of lib cannot be rendered due to missing conventions. + +### some multiline comments are doc-comments (implicitly) + +Currently many places (outside /lib) utilize the multiline comment `/* */` to write multiline comment documentation. +However this is also not consistent across nixpkgs and was never specified as the doc-comment format. + +- There is no respective single line comment, that would allow documentation rendering +- Inconsistent usage of multiline comments prevents us from using them directly. ### Differentiate from regular comments @@ -128,7 +136,7 @@ Other tools that work directly with the nix AST and comments: # Detailed design [design]: #detailed-design -**Proposed Solution:** +**Proposed Solution** Each line of a doc-comment starts with `##`. @@ -163,7 +171,7 @@ The following exampple demonstrates the concrete usage scenario. The following abstract rules describe how to write doc-comments. -### doc-comments are all comments. That start with `##` e.g. `## {content}` +### doc-comments are all comments. That start with and additional prefix `##` e.g. `## {content}` ```nix ## Documentation From acb9815fe207cfeadabd98bc957652e79f462df5 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Fri, 14 Apr 2023 14:56:59 +0200 Subject: [PATCH 047/110] Feedback from docs-team --- rfcs/0145-doc-strings.md | 561 ++++++++++++++++++--------------------- 1 file changed, 261 insertions(+), 300 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index a6b7deffb..16a716fa8 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -3,7 +3,7 @@ feature: doc-comment-standard start-date: 2023-03-27 author: hsjobeki co-authors: (find a buddy later to help out with the RFC) -shepherd-team: @DavHau, ? (If you are interested please PM the author on matrix or comment) +shepherd-team: (@DavHau, ... ? If you are interested, please PM the author on matrix or comment) shepherd-leader: (name to be appointed by RFC steering committee) related-issues: (will contain links to implementation PRs) --- @@ -13,7 +13,7 @@ related-issues: (will contain links to implementation PRs) Propose a standard format for Doc-comments. -This RFC includes two separate concerns: +This RFC includes two concerns that define a doc-comment: - Outer format rules to allow distinction between regular comments and doc-comments - Inner format rules that describe the required format of the content. @@ -23,18 +23,18 @@ However, both concerns relate closely to each other; We thought it made sense an # Motivation [motivation]: #motivation -The following are the envisioned goals +The following are the envisioned goals. - be able to generate documentation from code for any nix expression. - be able to distinguish between documentation-relevant comments and unrelated comments. - make doc comments easy to write and read - be able to parse and render doc comments nicely -- standardize a format for doc comments that can be extended by further RFCs +- standardize a format for doc comments that further RFCs can extend This RFC is a significant change to the existing documentation convention -but allows to distinguish between regular comments and doc comments. This is important because arbitrary code comments should not end up in generated documentation. +but allows distinguishing between regular and doc comments. Having distinction is essential because arbitrary code comments should not end up in generated documentation. -> Hint: Generating static documentation is controvert topic in nixpkgs. It was found that is impossible to generate accurate documentation statically. A correct solution would involve evaluation of expressions in some way. This already goes deeply into implementation details and is thus not further discussed in this document. Although we envision solutions to solve this issue. +> Hint: Generating static documentation is controvert topic in nixpkgs. We found that it is impossible to generate accurate documentation statically. A correct solution would involve the evaluation of expressions in some way. This already goes deeply into implementation details and is thus not discussed further in this document. However, we envision solutions to solve this issue. ## Current State @@ -43,7 +43,7 @@ Many inconsistently written comments document specific parts of nixpkgs and othe We use some of them to generate documentation automatically. (e.g., nixpkgs/lib via [nixdoc](https://github.com/nix-community/nixdoc) ) -This solution requires a lot of manual work; more specifically, *nixdoc* is a custom tool that works only for that purpose. +This solution requires much manual work; more specifically, *nixdoc* is a custom tool that works only for that purpose. Here is an example of the format understood by *nixdoc*: @@ -77,7 +77,7 @@ Those comments are only used and parsed consistently in the /lib folder. Outside In general, the format for writing documentation strings is **not specified**. -The *nixdoc*-tool enforces a somewhat consistent format but the actual format was never specified and is not enforced in parts of nixpkgs where nixdoc is currently not applied. +The *nixdoc*-tool enforces a somewhat consistent format, but the basic format was never specified and cannot be enforced in parts of nixpkgs where nixdoc is currently not applied. Extending the scope of *nixdoc* is not the primary goal. Instead, we should find formal rules for writing *doc-comments*. Tools like *nixdoc* can then implement against this RFC instead of the format relying on nixdoc implementation details. @@ -91,23 +91,23 @@ e.g., - file that exports a constant expression. - files outside of lib cannot be rendered due to missing conventions. -### some multiline comments are doc-comments (implicitly) +### Some multiline comments are doc-comments (implicitly) -Currently many places (outside /lib) utilize the multiline comment `/* */` to write multiline comment documentation. -However this is also not consistent across nixpkgs and was never specified as the doc-comment format. +Many places (outside /lib) currently utilize the multiline comment `/* */` to write multiline comment documentation. +However, this is also inconsistent across nixpkgs and was never specified as the doc-comment format. -- There is no respective single line comment, that would allow documentation rendering +- There is no respective single-line comment, that would allow documentation rendering - Inconsistent usage of multiline comments prevents us from using them directly. -### Differentiate from regular comments +### Impossible to differentiate from regular comments -The format doesn't allow any distinction between doc-comments or regular comments. +The format does not allow any distinction between doc-comments and regular comments. Having a distinction would allow us to 1. Find all comments that are part of the documentation 2. Render them in the documentation format -3. Connect the documentation to the exact places in the nix code. This is already done, but only for nixpkgs/lib. +3. Connect the documentation to the exact places in the Nix code. ### References to the problems above @@ -129,400 +129,361 @@ Having a distinction would allow us to Other tools that work directly with the nix AST and comments: -- [noogle](https://noogle.dev) - Nix API search engine. It allows to search functions and other expressions. +- [noogle](https://noogle.dev) - Nix API search engine. It allows searching functions and other expressions. - [nix-doc](https://github.com/lf-/nix-doc) - A Nix developer tool leveraging the rnix Nix parser for intelligent documentation search and tags generation - [manix](https://github.com/mlvzk/manix) - A fast CLI documentation searcher for nix. # Detailed design [design]: #detailed-design -**Proposed Solution** -Each line of a doc-comment starts with `##`. +Each subsection here contains a decision along with arguments and counter-arguments for (+) and against (-) that decision. -The content of all doc-comments is rendered using markdown. -Following the [commonmark-specification](https://spec.commonmark.org/) -## Doc-comments +## `/**` to start a doc-comment -The following exampple demonstrates the concrete usage scenario. +**Observing**: Doc-comments' outer format should be a distinctive subset of regular comments. Nevertheless, it should allow native writing without needing an ide or full-blown-language servers. -### A doc-comment referencing the subsequent expression +**Considering**: `/** {content} */` where `{content}` is the inner format which is discussed later. -`somefile.nix` - -```nix -{ - ## - ## Documentation for 'mapAttrs' - ## - ## # Example - ## - ## - ## - ## # Type - ## - ## - mapAttrs = f: s: <...>; -} -``` - -## Format Rules - -The following abstract rules describe how to write doc-comments. - -### doc-comments are all comments. That start with and additional prefix `##` e.g. `## {content}` - -```nix -## Documentation -## follows simple rules -## and can span multiple lines -``` - -### Doc-comments always document / relate to an expression +**Decision**: use `/** {content} */` as the outer format. -In detail: `##` relates to the next AST-Node. However technical details for tracking names and aliases is not part of this document. +
+Arguments -> Vision: Implement a custom evaluator that specializes in tracking references of doc-comments within the nix expression tree. This is however a technical concrete solution that may be build after this rfc is accepted. - -### Doc-comments continue with `##` +- (+) Stays mostly compatible with the currently used multiline comments. +- (+) Is a strict subset of multiline comments, which allows multiline documentation natively. +- (+) Does not need Language Server (LSP) support for productive usage. (In contrast to e.g., `##`) +- (+) Allows copy-pasting content without the need for re-formatting. +- (-) Is visually less distinctive. +- (-) Indentation is more complex and visually less present. + - (+) If its indentation logic implements nix's magic multiline strings, it would be most intuitive. +- (-) Takes up more vertical space +- (+) Takes up less horizontal space. (In contrast to e.g., `##` lines don't have to be prefixed) +- (+) Only one single character to change the semantics of the whole comment. + - (+) Allows for quickly adding and removing things from the documentation. + - (-) Accidentally adding/removing could happen. + +
-Doc-comments start with `##` (leading whitespaces are allowed). -The docstring is continued in the following line if it also starts with `##`. -Leading whitespace is allowed. +## CommonMark as the content of doc-comments -Example: docstring continuation +**Observing**: Doc-comments' content should be intuitive to read and write and straightforward to render. The nixdoc convention is not widely adopted outside certain places (e.g /lib) in nixpkgs. This may also come from a lack of understading the current self-cooked format. -```nix -## doc-comment A -## .... -## This block has no expression in the following line. -## Therefore, it doesn't have any effect -## .... -## doc-comment A - - -## doc-comment B -## -- This block documents the purpose of '1' -## doc-comment B -1 -``` +**Considering**: CommonMark as the content format. -> Hint: It is common to use the same indentation as the referenced expression. - -```nix - ## The function foo makes us happy - foo = arg: #... -``` +**Decision**: CommonMark is the content of all doc-comments. -### The content of a doc-comment is Markdown +
+Arguments -The content of a doc-comment is parsed using Markdown following the [commonmark specification](https://commonmark.org/). Thus it allows for intuitive usage without knowledge of complex syntax rules. +- (+) CommonMark is the official format in nix; Decided in [RFC72](https://github.com/NixOS/rfcs/blob/master/rfcs/0072-commonmark-docs.md). + - (+) It Would be consistent if this RFC builds upon the previous one. + - (+) Further Arguments for CommonMark, in general, can be found in RFC72 +- (+) Allows copy-paste from and to markdown files. Allowing easy refactoring if documentation grows and needs to be splitted in separate files. +- (-) Strictly binding doc-comments' content to commonMark might restrict users. + - (+) Users/Tools can still use regular comments or develop alternative solutions. +- (-) CommonMark does not specify the current rich features from nixdoc such as predefined sections and structures that could be used as source for automated toolings. Such as types for type-checking or examples to run automated-tests + - (+) future tools can still build their conventions on top of this RFC. They might not directly specify them in an RFC but instead, be a tool developers choose for a specific codebase. However, we have yet to get that tools. So it is good when rich features remain unspecified. + +> Markdown is the most straightforward and most accepted format +> for writing and rendering documentation. + +
-Top level H1 headings, starting with a single `#`, indicate sections. Sections might be specified in future rfcs. +## Place doc-comments before the referenced expression -Common Sections: +**Observing**: Doc-comments currently are placed above the expression they document. More precisely, currently, only named attribute bindings `foo = ` can be documented. There is also the need to support anonymous functions where they are implemented, not where they get names. More generally, there is the need to document unnamed expressions. -- `# Examples` -- `# Type` +**Considering**: General reference logic between doc-comments and expressions. -Sections that might be specified in future rfcs: +**Decision**: Doc-comments always relate to the expression in the next AST-Node. -- `# Type` -- `# Arguments` -- `# Meta` +
+Arguments -Until they are actually specified. +- (+) Doc-comments should have only one variant if possible to reduce complexity. Referencing the next node seems straight forward. +- (-) A variant that references the pevious node in the AST should be avoided if possible to reduce complexity. +- (+) Relation between documentation and the referenced implementation is clear and back-trackable. +- (-) Concrete Implementation might be complex. +- (-) Unclear if complete documentation might also need backwards references. + - (-) e.g. rust uses backwards references only at top of file comments. + - (+) Nix files can only have ONE expression, the next AST Node in case of top-of-file comments is thus always only that one expression. (Unlike in Rust) +- (-) Tools need to be smart enough to understand asignments `=` and other forms of creating names for anonymous expressions. (e.g `callPackage` and `import` ) + - (+) Tools can still come up with other solutions, that dont involve calculating everything dynamically from nix code, but could also involve a static configuration. + - (+) The whole `tool point` is an implementation detail. As long as it is not impossible to implement. The current tool `nixdoc` already proves that it is possible to have static documentation to a certain degree. -Future RFCs may specify sub-headings of `# Meta`. Its usage is reserved. +
-## Keywords -[keywords]: #keywords +## Doc-comment examples -The following keywords start reserved markdown sections +The following examples demonstrates the concrete usage scenarios: -> We wanted to keep the list of initial keywords short. So by the time this RFC focuses on the formal aspects of doc-comments first. More keywords and features for them may be added later on. +`somefile.nix` -| Keyword | Description | Note | -| --- | --- | --- | -| `# Examples` | Starts the Example-block. Often contains comprehensive code examples | | -| `# Type` | Start the Type-block; Just any free text; but we recommend following the existing convention of the current `Type:` field | Syntax may eventually be specified in the future. | -| `# Meta` | Under this section future rfcs may specify their sub-sections. | Sub-sections within meta will avoid namespace collisions in future RFCs | -| `# Arguments` | Needed for describing the functions arguments. Is just any free text until specified | | +~~~nix +{ + /** + Documentation for mapAttrs + + # Example + + ``` + mapAttrs {attr = "foo"; } ... + ``` + + # Type + + ``` + mapAttrs :: a -> b -> c + ``` + + */ + mapAttrs = f: s: <...>; +} +~~~ -## Why change the existing section specifiers? +## Examples -First, there are no actual block specifiers within nix or nixpkgs. The existing blocks heavily depend on a tool called `nixdoc` and not vice versa. +This section contains examples for the different use-cases we would end up with; Visualize them and emphasize the previously discussed characteristics. -> `nixdoc` MUST be changed to support this RFC. (See [Future work](#future-work)) +### Attribute bindings -The sequence `Example:` has some drawbacks when it comes to syntax: +`somefile.nix` -1. It is possible that this sequence occurs in a natural text without the intention to start a new doc-comment section. -2. It doesn't visually stand out. -3. It is terrible that the line needs to start with `Example:` to be valid syntax. However, it is a good practice while writing comments; it should be optional. -4. It neither follows the `@param` (c/c++/java,...) convention nor the markdown headings convention (rust); instead is nixdoc-home-cooked. +~~~nix +{ + /** + mapAttrs is a well known function + + # Examples + + ```nix + some code examples + ``` + + */ + mapAttrs = f: s: #... +} +~~~ + +### NixOS Module documentation + +> This is an example of what is possible. The future conventions may still evolve from the nix-community + +NixOS Modules also can produce documentation from their interface declarations. However, this does not include a generic description and usage examples. + +`myModule.nix` +~~~nix +/** + This is a custom module. + + It configures a systemd service + + # Examples + + Different use case scenarios + how to use this module + +*/ +{config, ...}: +{ + config = f: s: f s; +} +~~~ + +### Anonymous function documentation + +> This is an example of what is possible. The future conventions may still evolve from the nix-community + +`function.nix` +~~~nix +/** + This is an anonymous function implementation. + + It doesn't have a name yet. + But documentation can be right next to the implementation. + + The name gets assigned later: + + ```nix + { + plus = import ./function.nix; + } + ``` + + (future) native nix or community tools provide + implementations to track doc-comments within the nix evaluator. + Documentation of `sum` can then be inferred. + This still needs to be specified/implemented! + +*/ +{a, b}: +{ + sum = a + b; +} +~~~ -## Interactions +### Anonymous expression documentation -doc-comments can be attached to AST nodes without affecting the actual compile-, evaluation- or build-time because they are just comments. Specialized tools can handle those comments and create static documentation from them. Also, integration with LSP is possible. (See [@flokli's nix lsp-whishlist](https://hackmd.io/@geyA7YL_RyiWJO6d5TbC-g/Sy6lVrgW3) for inspirations) +> This is an example of what is possible. The future conventions may still evolve from the nix-community -Following this RFC means refactoring for existing comments, but it also means that **we can finally use all comments (automated!) that were intended to be doc-comments** +`exp.nix` +~~~nix +/** + This is an anonymous string. + + Its purpose can be documented. + + Although this example is quite superficial, there might be use cases. +*/ +"-p=libxml2/include/libxml2" +~~~ # Drawbacks [drawbacks]: #drawbacks -Drawbacks of this rfc. -- Changes the existing comments inside the code base. +## Changes the existing comments inside the code base. -This could mostly be automated. (e.g via codemod) +This could mainly be automated. (e.g. via codemod) -Also, this affects only the `lib` folder and a few other places that are currently used to build the documentation. +Also, this affects only the `lib` folder and a few other places currently used to build the documentation. -# Alternatives -[alternatives]: #alternatives - -While designing this RFC multiple alternative formats where considered. They can be found in the following section to understand the overall decisions that where made in the sections earlier. - -In general, we needed the following: +## Changes in the `nixdoc`tooling is required. -1. General format for doc-comments. -2. Format for headings and the allowed content. - -> It would be nice if this could be close to the markdown format. -> Markdown is the most straightforward and most accepted format -> for writing and rendering documentation. +It remains an open question that needs to be tried; -## General Formats +Can we still build the current nixos manuals with the new standard? -| Property / Approach | `##` | `/** */` | `Javadoc` | -|---|---|---|---| -| Inspired by | Rust | Current nixpkgs.lib | C++/Java/Javascript | -| Changes the existing code by | Much | Less | Even More | -| Needs Termination | No | Yes | Yes | -| Indentation | Clear | Poor | Poor | -| Needs vertical space | No | Yes | Yes | -| Visual distinction from comments | High | Low | Medium | -| Needs Autocompletion (Language Server) to continue the next line. | Yes | No | Yes | -| Punctuation Variations / Amount of different special characters | Less | More | More | -| Markdown compatibility (also depends on indentation clarity) | Good, but visual conflicts with headings` #` | Poor | Medium | -| breaks when interrupted with newlines | Yes | No | ? | +- We can wait to change everything. +- Migration can happen in small steps. -**Proposed format:** - -Use `##` to start a doc-comment. This allows clear visual separation from regular comments. -And provides a good compatibility with the strived markdown content. +# Alternatives +[alternatives]: #alternatives +## All considered outer formats + +| Property / Approach | `##` | `/** */` | `Javadoc` | `/*\|` or `/*^` | +|---|---|---|---|---| +| Inspired by | Rust | Current nixpkgs.lib | C++/Java/Javascript | Haskell Haddock | +| Changes the existing code by | Much | Less | Even More | Less | +| Needs Termination | No | Yes | Yes | Yes | +| Indentation | Clear | like Nix's multiline strings, thus **Intuitive** | Poor | ? | +| Needs vertical space | No | Yes | Yes | Yes | +| Visual distinction from comments | High | Low | Medium | Medium | +| Needs Autocompletion (Language Server) to continue the next line. | Yes | No | Yes | No | +| Punctuation Variations / Amount of different special characters | 1 (Less) | 2 (Medium) | 2 (Medium) | 3 (More) | +| Markdown compatibility (also depends on indentation clarity) | Good, but visual conflicts with headings` #` | Good | Medium | Good | +| breaks when interrupted with newlines | Yes | No | ? | No | +| Simplicity (Brainload) | Medium | Simple | Complex | More Complex | + ### Refactoring note - + +**Observing**: From a refactoring perspective, it might also be interesting to see how many conflicts the different formats would cause. + nixpkgs comments: -- `##` ~4k usages (most of them are used for visual separation e.g. `###########`) +- `##` ~4k usages (most of them for visual separation e.g., `###########`) - `#` ~20k usages - `/*` ~6k usages -- `/**` 160 usages (most of them are completely empty ?) +- `/**` 160 usages (most empty ?) + +Choosing `/**` or subsets would cause minor conflicts within current nixpkgs. While this is NOT the main reason for the final decision, it MUST be considered. -## General Headings -| Property / Approach | `# ` | `@:` | -|---|---|---| -| Inspired by | Markdown | Doxygen | -| Changes the existing code by | Minor | Minor | -| Needs vertical space | Recommended | No | -| Visual distinction from comments | Low | High | -| Markdown compatibility | Best | None | +## Just free text as a content format -**Proposed headings:** +While this allows the most freedom, it is usually considered the best option, not creating any restrictions. -Use markdown headings `# `. This allows best compatibility with the already specified markdown/commonmark format. Allowing for easy and intuitive usage for comments. +But nix/rfc72 defines commonMark as the official documentation format. +This is why we decided to follow this convention. -## Consequences to not implementing this +## Consequences of not implementing this - By not implementing this feature, nix gains no ability for tool-generated documentation. -- Documentation will be defined by nixdoc, not by the nix community itself. +- Documentation will be defined by nixdoc, not by the nix community. - Many existing comments written for documentation will remain un-discoverable. -## Enforcing lsp support is not great - -Single-line comments `##` require using a Language Server (LSP) to make developers work productively. It is desirable to have a doc-comment format that works nicely without an LSP. To achieve this with a variant of the multiline comment `/**{content}*/` as the format for doc-comments to be the respective subset of multiline comments - -## Examples - -This section contains examples for the different formats to visualize them and emphasize the previously discussed characteristics. - -### `##` inspired from rust's `///` - -Example of the proposed `##` format; with `Markdown` Headings. - -`somefile.nix` - -```nix - ## - ## - ## # Example - ## - ## - ## - ## # Type - ## - ## - mapAttrs = f: s: #... -``` - -### `/** */` inspired by the current multiline strings - -Example of an alternative multiline`/** */` format; with `Markdown` Headings. - -TODO: Work out the details for multiline doc comments together with documentation-team. This section might be removed and included into the "detailed design" section. - -```nix - /** - mapAttrs is a function in lib - - # Example - - ``` - comprehensive example - shows how to use this - ``` - - # Type - - foo -> bar - */ - mapAttrs = f: s: #... -``` - - -## Javadoc style - -Example of the discarded `javadoc` format. - -```java - /** - * A short description - * @author Stefan Schneider - * @version 1.1 - * @see https://some.url - */ - public class Product { - ... - } -``` - -## Alternative approach - just comments - -There is the idea from python that doc-comments are just strings, not even special ones. Strings will be docstrings if they follow specific placement rules. However, we thought this was a bad idea to follow. Such complex placement rules require the users to understand where those places are; with nix syntax, this is slightly more complex than with python. Because we don't have keywords such as `class MyClass():` or `def function():` where placement would be obvious - # Unresolved questions [unresolved]: #unresolved-questions -- `nixodc` offers comments to describe function arguments. This is currently not compatible until some sections for `args` are defined. +- `nixodc` offers comments to describe function arguments, and this is currently discarded once they are directly described in prose by the user. -- Will `nix` itself implement native support like in rust -> `cargo doc` +- Will `nix` itself implement native support like in rust -> `cargo doc`? - How can a tool keep the connection from where a docstring was defined and where the attribute was exposed (lib/default. nix exposes mapAttrs which is defined at lib/attrsets.nix) - There are more complicated things. --> Answer: A Tool might be able to keep track of a percentage of expressions, and sometimes it may be very hard or impossible. For that case, the doc-comment can offer a dedicated Keyword to override the scope. - -e.g. - -The following is an idea for a problem that will arise if tools try to track doc-comments' positions and the location in the nixpkgs tree. (Although this problem is not nixpkgs specific) - -```nix -## This file is called somewhere that cannot be automatically tracked/is impossible to analyze statically. -## The 'TreePath' override can be used by the docstring author to set fixed paths in the nixpkgs expression. -## This tells the doc-tool that the expression is available in two aliases: `pkgs.stdenv` and `lib.sth` -## (This behavior will not be specified in this RFC) -## # Meta -## ## TreePath -## - pkgs.stdenv -## - lib.sth - -{...}: -{ - # returns something -} -``` +**Answer**: This is a tooling question; implementation details won't be discussed in detail. It is possible to track comments in the AST. As shown before in this PR: [nix/#1652](https://github.com/NixOS/nix/pull/1652) # Future work [future]: #future-work ## Editor support -When starting hitting {enter} inside a doc-block the new line, should be automatically prefixed with `##` accordingly. -This is done in rust similarly. Nix already offers a bunch of LSP's e.g. [nil](https://github.com/oxalica/nil), [rnix-lsp](https://github.com/nix-community/rnix-lsp) are the most common ones. - -- Those LSP's should implement the simple "line continuation" feature. (I don't know the exact name here) - Implement displaying the related documentation when hovering over an expression. (lspAction/hover) -## Nixodc - -Nixdoc needs to be changed in order to differentiate between regular comment and doc-comments. -There might be an intermediate phase of transition, where the old syntax and features is supported for a while. +Nix already offers a bunch of LSP's e.g. [nil](https://github.com/oxalica/nil), [rnix-lsp](https://github.com/nix-community/rnix-lsp) are the most common ones. -- When extending nixdoc or writing dedicated parsers, the following persons can assist: [@hsjobeki] -## Documentation generators - -Generating documentation from doc-comments is still a challenge. -If we'd like the fully automated approach, we definitely need something that can also evaluate nix expressions. -(We have such a module in `tvix` which needs to be investigated more here) - -Alternatively we can use the future nixdoc together with a static `map.json` that contains the scope for every discovery file/path in nixpkgs. - -As this second approach is much easier; We propose this is how we should initially start to extend the scope. +## Nixodc -## More specialized section headings +Nixdoc must be changed to differentiate between regular comments and doc-comments. +There might be an intermediate phase of transition, where the old syntax and features are supported in parallel to allow a phase of transition and refactoring of existing documentation comments. -### Type +## Documentation generators -- An RFC under construction specifies the used syntax within the `Type`-Block. It depends on this RFC, as it is the groundwork to provide a standardized field where additional rules can apply. +- Future nixdoc could have a static `map.json` that contains the scope for every discovery file/path in nixpkgs. -## Native support in nix +As this second approach is much easier, We propose this is how we should initially start to extend the scope. -- `NixOS/nix` should implement native support for doc-comments so that our users don't have to rely on nixpkgs or external tools. Those tools can still exist and provide more custom functionality, but documenting nix expressions should be natively possible. +Generating documentation from doc-comments dynamically would still be excellent but remains a challenge. +If we want the fully automated approach, we need something to evaluate nix expressions. -## Provide a stable and reliable format +We solve such concerns in `tvix` or in `nix`, which could provide a `nix-analyzer` that pre-evaluates expressions and their respective documentation. -- Every existing and future tool can implement against this RFC and rely on it. +## Type -# Related Tools +- An RFC under construction specifies the used syntax within the `# Type` Heading. +- The `type` feature should belong in the nix syntax. Try them within the comments first; This is still possible. -- [Rustdoc](https://doc.rust-lang.org/rustdoc/how-to-write-documentation.html) -- [Nixdoc](https://github.com/nix-community/nixdoc) +- see a [preview](https://typednix.dev) of an eventual future doc-type-syntax. -## Further +## Native support in Nix -There is an actual but rather old PR that uses just comments to show documentation in the nix repl for functions. +- `NixOS/nix` should implement native support for doc-comments so that our users do not have to rely on nixpkgs or external tools. Those tools can still exist and provide more custom functionality, but documenting nix expressions should be natively possible. -- https://github.com/NixOS/nix/pull/1652 +# References -We envision gradual type checking for nix. +## Other Conventions -A weak source of type constraint could be the `# Type` field in doc-comments until nix may introduce its own native type system. -Very concrete doc-typing-syntax may allow gradual type checking. +- [Rust](https://doc.rust-lang.org/stable/reference/comments.html#doc-comments) +- [Python](https://peps.python.org/pep-0257/) +- [JSDoc](https://jsdoc.app/) -- see a [preview](https://typednix.dev) of an eventual future doc-type-syntax. +## Related tools -## People discussed with +- [Nixdoc](https://github.com/nix-community/nixdoc) +- [Rustdoc](https://doc.rust-lang.org/rustdoc/how-to-write-documentation.html) +- [Doxygen](https://www.doxygen.nl/) -> People mentioned here might be not yet aware of this rfc. -> I'll ping them in the next few days to make sure they are okay with being mentioned here. +## Related discussions -About doc-comments/doc-comments in general +### About doc-comments/doc-comments in general. +- [@documentation-team (meet @ 13.Apr 2023)](https://discourse.nixos.org/t/2023-04-13-documentation-team-meeting-notes-41/27264) - [@flokli](https://github.com/flokli) - one of the [tvix](https://tvl.fyi/blog/rewriting-nix) authors - [@tazjin](https://github.com/tazjin) - Original Author of `nixdoc`, one of the `tvix` authors -About documentation approaches on independent frameworks +- There is an actual but rather old PR (@roberth) that uses just comments to show documentation in the nix repl for functions. -> https://github.com/NixOS/nix/pull/1652 + +### About documentation approaches on independent frameworks. - [@davHau](https://github.com/davHau) - Author of [dream2nix](https://github.com/nix-community/dream2nix), (And many other nix-frameworks) -About defining weakly typed-interfaces for nix with doc-comments +### About using comments as weak sources for typed nix. -- [@roberth](https://github.com/roberth) - nixpkgs Architecture Team +- [@roberth](https://github.com/roberth) - nixpkgs Architecture Team / nix core team ? - [@aakropotkin](https://github.com/aakropotkin/) - nixpkgs Architecture Team From 86475deb6953cdc4e40690d88dc0227c57bdd938 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Mon, 17 Apr 2023 09:39:48 +0200 Subject: [PATCH 048/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 247 ++++++++++++++++++++++----------------- 1 file changed, 138 insertions(+), 109 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 16a716fa8..c47a61974 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -3,7 +3,7 @@ feature: doc-comment-standard start-date: 2023-03-27 author: hsjobeki co-authors: (find a buddy later to help out with the RFC) -shepherd-team: (@DavHau, ... ? If you are interested, please PM the author on matrix or comment) +shepherd-team: (@DavHau; tbd.) shepherd-leader: (name to be appointed by RFC steering committee) related-issues: (will contain links to implementation PRs) --- @@ -11,39 +11,39 @@ related-issues: (will contain links to implementation PRs) # Summary [summary]: #summary -Propose a standard format for Doc-comments. +Propose a standard format specification for Doc-comments. This RFC includes two concerns that define a doc-comment: - Outer format rules to allow distinction between regular comments and doc-comments - Inner format rules that describe the required format of the content. -However, both concerns relate closely to each other; We thought it made sense and reduced bureaucracy to address that in a single RFC. +However, both concerns relate closely to each other; It makes sense and reduces bureaucracy to address that in a single RFC. # Motivation [motivation]: #motivation The following are the envisioned goals. -- be able to generate documentation from code for any nix expression. -- be able to distinguish between documentation-relevant comments and unrelated comments. -- make doc comments easy to write and read -- be able to parse and render doc comments nicely -- standardize a format for doc comments that further RFCs can extend +- Be able to generate documentation from code for any nix expression. +- Be able to distinguish between documentation-relevant comments and unrelated comments. +- Make doc comments easy to write and read +- Be able to parse and render doc comments nicely +- Standardize a format for doc comments that further RFCs can extend -This RFC is a significant change to the existing documentation convention -but allows distinguishing between regular and doc comments. Having distinction is essential because arbitrary code comments should not end up in generated documentation. +This RFC is a significant change to existing documentation conventions. +It allows distinguishing between regular and doc comments. Having distinction is essential because arbitrary code comments should not end up in generated documentation. -> Hint: Generating static documentation is controvert topic in nixpkgs. We found that it is impossible to generate accurate documentation statically. A correct solution would involve the evaluation of expressions in some way. This already goes deeply into implementation details and is thus not discussed further in this document. However, we envision solutions to solve this issue. +> Hint: Generating static documentation is controvert topic in nixpkgs. We found that it is impossible to generate accurate documentation statically. A correct solution would involve the evaluation of expressions in some way. ## Current State -We currently utilize a `doc-comment`-like functionality to build a subset of static documentation for nix functions. (e.g., nixpkgs.lib documentation via: [nixdoc](https://github.com/nix-community/nixdoc)) -Many inconsistently written comments document specific parts of nixpkgs and other nix-frameworks (see [references-to-this](#references-to-the-problems-above)). +Currently, Nix does not specify doc comments. Over time there evolved certain conventions, But the nix-community never specified those. +Within this RFC, we want to clarify which convention we should use, harmonize and improve the format for doc-comments to achieve the envisioned goals. -We use some of them to generate documentation automatically. (e.g., nixpkgs/lib via [nixdoc](https://github.com/nix-community/nixdoc) ) +We use a tool called [nixdoc](https://github.com/nix-community/nixdoc) to use a `doc-comment'- like functionality, which allows us to build a subset of static documentation of the `nixpkgs.lib` functions. -This solution requires much manual work; more specifically, *nixdoc* is a custom tool that works only for that purpose. +The format of a doc-comment was, however, never specified. The nix community does not understand the rules enforced by `nixdoc`, which we believe is one of the reasons for its lack of adoption. Here is an example of the format understood by *nixdoc*: @@ -69,11 +69,9 @@ Here is an example of the format understood by *nixdoc*: ## Current problems -### Inconsistent usage outside /lib folder +### Lack of having an agreed Format -Those comments are only used and parsed consistently in the /lib folder. Outside of this folder the format doesn't follow the convention strictly. Also the comments outside /lib are not used to generate any output. - -### Unspecified format +The Nix community interprets the outer format and content of doc-comments differently. Within nixpkgs, several conventions and patterns still need to be harmonized. Some are compatible with nixdoc, while others do not and decide upon their conventions. We immediately observe one clear thing: Most of them utilize some form of markdown, even if the content is not rendered. In general, the format for writing documentation strings is **not specified**. @@ -81,27 +79,9 @@ The *nixdoc*-tool enforces a somewhat consistent format, but the basic format wa Extending the scope of *nixdoc* is not the primary goal. Instead, we should find formal rules for writing *doc-comments*. Tools like *nixdoc* can then implement against this RFC instead of the format relying on nixdoc implementation details. -### Only specific placements work - -The placement of those comments requires precisely commenting at the attribute set containing the function declarations, which is not usable for general-purpose documentation strings. - -e.g., - -- file that directly exports the lib-function without wrapping it in an attribute set. -- file that exports a constant expression. -- files outside of lib cannot be rendered due to missing conventions. - -### Some multiline comments are doc-comments (implicitly) - -Many places (outside /lib) currently utilize the multiline comment `/* */` to write multiline comment documentation. -However, this is also inconsistent across nixpkgs and was never specified as the doc-comment format. - -- There is no respective single-line comment, that would allow documentation rendering -- Inconsistent usage of multiline comments prevents us from using them directly. - ### Impossible to differentiate from regular comments -The format does not allow any distinction between doc-comments and regular comments. +The format does not allow any distinction between doc comments and regular comments. Having a distinction would allow us to @@ -117,32 +97,60 @@ Having a distinction would allow us to - [trivial-builders](https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/trivial-builders.nix) - [stdenv/mkDerivation](https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix) - [nixos/lib/make-disk-image](https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-disk-image.nix) -- more... +- more #### frameworks - [dream2nix/utils](https://github.com/nix-community/dream2nix/blob/main/src/utils/config.nix) - [dream2nix/templates/builder](https://github.com/nix-community/dream2nix/blob/main/src/templates/builders/default.nix) -- more... +- more -#### Other tools +#### Current tools -Other tools that work directly with the nix AST and comments: +Other tools that work directly with the nix AST and comments. +Those would strategically benefit from this RFC, even if it means refactoring and migration efforts. - [noogle](https://noogle.dev) - Nix API search engine. It allows searching functions and other expressions. - [nix-doc](https://github.com/lf-/nix-doc) - A Nix developer tool leveraging the rnix Nix parser for intelligent documentation search and tags generation -- [manix](https://github.com/mlvzk/manix) - A fast CLI documentation searcher for nix. +- [manix](https://github.com/mlvzk/manix) - A fast CLI documentation searcher for Nix. # Detailed design [design]: #detailed-design - Each subsection here contains a decision along with arguments and counter-arguments for (+) and against (-) that decision. +> Note: This RFC does not promote any tool; It focuses only on the format, not implementation. At the same time, this RFC must be technically feasible. + +## Discontinue nixdoc's custom format + +**Observing**: Nixdoc currently provides some format that can generate documentation from a small fraction of files in nixpkgs, Namely the files for `.lib` functions. The folder provides a consistent community convention that allows the building of static documentation. Applying nixdoc to files outside that directory is only partially possible due to the lack of a standard doc-comment format. + + +**Considering**: Should we propose the `nixdoc` format as a community standard? + +**Decision**: Discontinue nixdoc's custom format in favor of using something more generic. + +
+Arguments + +- (+) Stays primarily compatible with the currently used comments in lib. + - (-) Outside the ./lib folder, nixodc is not a convention. +- (-) Is not widely understood. +- (-) Is not widely used. (e.g., Outside the /lib folder) +- (-) The format is specific, should be more generic, and follow established conventions. (e.g., Markdown, Doxygen, JSdoc, Haddock) +- (-) The format is too complex and needs a whole list of complex rules to describe the exact format and content rules; this can also be observed in many false-formatted comments that do not render correctly. + - (-) The sections are implicitly continued. + - (-) The predefined section headers/keywords are very close to natural language and may accidentally be used. + - (-) Syntax to start new sections does not follow conventions (e.g., `Example:`) In contrast JSdoc defines `@Example`; Markdown `# Example`; Doxygen `@example`; Haskell-Haddock `>>>` + - (-) The content is implicitly rendered differently depending on predefined rules. Those rules need to be understood/feel natural. +- (+) It Is already quite established and has some adoptions outside the `/lib` folder. +- (-) Some multiline comments become doc-comments, but it is unclear which comments have such a property. + +
## `/**` to start a doc-comment -**Observing**: Doc-comments' outer format should be a distinctive subset of regular comments. Nevertheless, it should allow native writing without needing an ide or full-blown-language servers. +**Observing**: Doc-comments' outer format should be a distinctive subset of regular comments. Nevertheless, it should allow native writing without an ide or full-blown-language servers. **Considering**: `/** {content} */` where `{content}` is the inner format which is discussed later. @@ -151,17 +159,17 @@ Each subsection here contains a decision along with arguments and counter-argume
Arguments -- (+) Stays mostly compatible with the currently used multiline comments. +- (+) It is mostly compatible with the currently used multiline comments. - (+) Is a strict subset of multiline comments, which allows multiline documentation natively. -- (+) Does not need Language Server (LSP) support for productive usage. (In contrast to e.g., `##`) +- (+) Does not need Language Server (LSP) support for productive usage. (In contrast to, e.g., `##`) - (+) Allows copy-pasting content without the need for re-formatting. - (-) Is visually less distinctive. - (-) Indentation is more complex and visually less present. - - (+) If its indentation logic implements nix's magic multiline strings, it would be most intuitive. + - (+) It would be most intuitive if its indentation logic implements Nix's magic multiline strings. - (-) Takes up more vertical space -- (+) Takes up less horizontal space. (In contrast to e.g., `##` lines don't have to be prefixed) -- (+) Only one single character to change the semantics of the whole comment. - - (+) Allows for quickly adding and removing things from the documentation. +- (+) Takes up less horizontal space. (In contrast to, e.g., `##` lines do not have to be prefixed) +- (+) Only one character to change the semantics of the whole comment. + - (+) Allows to add and remove things from the documentation quickly. - (-) Accidentally adding/removing could happen.
@@ -169,56 +177,85 @@ Each subsection here contains a decision along with arguments and counter-argume ## CommonMark as the content of doc-comments -**Observing**: Doc-comments' content should be intuitive to read and write and straightforward to render. The nixdoc convention is not widely adopted outside certain places (e.g /lib) in nixpkgs. This may also come from a lack of understading the current self-cooked format. +**Observing**: Doc-comments' content should be intuitive to read and write and straightforward to render. The nixdoc convention is only widely adopted in certain places (e.g., /lib) in nixpkgs. It may also come from a need for more understanding of the current self-cooked format. **Considering**: CommonMark as the content format. **Decision**: CommonMark is the content of all doc-comments. +> Markdown is the most accepted and understood format for writing documentation in tech projects. Also, following the existing RFC-72 is highly profitable and consistent for the Nix ecosystem. +
Arguments - (+) CommonMark is the official format in nix; Decided in [RFC72](https://github.com/NixOS/rfcs/blob/master/rfcs/0072-commonmark-docs.md). - (+) It Would be consistent if this RFC builds upon the previous one. - (+) Further Arguments for CommonMark, in general, can be found in RFC72 -- (+) Allows copy-paste from and to markdown files. Allowing easy refactoring if documentation grows and needs to be splitted in separate files. -- (-) Strictly binding doc-comments' content to commonMark might restrict users. +- (+) Allows copy-paste from and to markdown files. We allow easy refactoring if documentation arises and needs to be split into separate files. +- (-) Strictly binding doc comments content to commonMark might restrict users. - (+) Users/Tools can still use regular comments or develop alternative solutions. -- (-) CommonMark does not specify the current rich features from nixdoc such as predefined sections and structures that could be used as source for automated toolings. Such as types for type-checking or examples to run automated-tests - - (+) future tools can still build their conventions on top of this RFC. They might not directly specify them in an RFC but instead, be a tool developers choose for a specific codebase. However, we have yet to get that tools. So it is good when rich features remain unspecified. - -> Markdown is the most straightforward and most accepted format -> for writing and rendering documentation. - +- (-) CommonMark does not specify the current rich features from nixdoc, such as predefined sections and structures that could be used as a source for automated toolings. Such as types for type-checking or examples to run automated-tests + - (+) Future tools can still build their conventions on top of this RFC. They might not directly specify them in an RFC but be a tool developers choose for a specific codebase. However, we have yet to get that tools. So it is good when rich features remain unspecified. +
## Place doc-comments before the referenced expression -**Observing**: Doc-comments currently are placed above the expression they document. More precisely, currently, only named attribute bindings `foo = ` can be documented. There is also the need to support anonymous functions where they are implemented, not where they get names. More generally, there is the need to document unnamed expressions. +**Observing**: Doc-comments currently are placed above the expression they document. More precisely, only named attribute bindings `foo = ` can be documented. There is also the need to support documentation for anonymous functions. More generally, there is the need to document anonymous expressions where they are defined. **Considering**: General reference logic between doc-comments and expressions. -**Decision**: Doc-comments always relate to the expression in the next AST-Node. +**Decision**: Doc-comments always relate to the expression in the next Abstract Syntax Tree Node (AST). + +> Note: Precedence is unnecessary as the AST already decided upon that while parsing. In terms of precedence, a doc-comment virtually always has the lowest possible value (e.g., 1). However, this RFC will not define precedence. +> +> Also Tool specific implementation questions must be resolved separately. Future RFCs to restrict or alter the behavior specified by this decision are valid if they present rational reasoning.
Arguments -- (+) Doc-comments should have only one variant if possible to reduce complexity. Referencing the next node seems straight forward. -- (-) A variant that references the pevious node in the AST should be avoided if possible to reduce complexity. -- (+) Relation between documentation and the referenced implementation is clear and back-trackable. +- (+) Doc-comments should have only one variant to reduce complexity. Referencing the next node seems straightforward. +- (-) A variant that references the previous node in the AST should be avoided to reduce complexity. +- (+) Relation between documentation and the referenced implementation is straightforward and back-trackable. - (-) Concrete Implementation might be complex. -- (-) Unclear if complete documentation might also need backwards references. - - (-) e.g. rust uses backwards references only at top of file comments. - - (+) Nix files can only have ONE expression, the next AST Node in case of top-of-file comments is thus always only that one expression. (Unlike in Rust) -- (-) Tools need to be smart enough to understand asignments `=` and other forms of creating names for anonymous expressions. (e.g `callPackage` and `import` ) - - (+) Tools can still come up with other solutions, that dont involve calculating everything dynamically from nix code, but could also involve a static configuration. - - (+) The whole `tool point` is an implementation detail. As long as it is not impossible to implement. The current tool `nixdoc` already proves that it is possible to have static documentation to a certain degree. +- (-) Unclear if complete documentation might also need backward references. + - (-), e.g., rust uses backward references only at the top of file comments. + - (+) Nix files can only have ONE expression; the next AST Node, in case of top-of-file comments, is thus always only that one expression. (Unlike in Rust) +- (-) Tools need to be smart enough to understand asignments `=` and other forms of creating names for anonymous expressions. (e.g., `callPackage` and `import` ) + - (+) Tools can still come up with other solutions that do not involve calculating everything dynamically from nix code but could also involve a static configuration. + - (+) The whole `tool point` is an implementation detail as long as it is not impossible. The current tool, `nixdoc`, already proves that it is possible to have static documentation to a certain degree. + +
+ +## Single doc-comments (do not exist) + +**Observing**: Nix offers two variants of comments; single- and multi-line comments. There may be use cases where it is desirable to have a form of single-line comments subtyped for doc-comment purposes. + +**Considering**: Single-line comment for documentation. + +**Decision**: Single-line comments **cannot be used** in any form for documentation puposes. + +> This decision was not easy and may still be controversial + +
+Arguments + +- (-) Doc-comments should have only one variant to reduce complexity. +- (-) documentation will likely take up more than one line. +- (-) If documentation grows bigger than one line, refactoring into a multiline-doc-comment must occur. +- (+) It Would be consistent with providing variants for both nix comments. +- (+) Offer the choice. +- (o) Single lines can also be concatenated to form multi-line documentation. +- (+) Takes up less vertical space +- (-) Visually confusing when every line starts with a reserved character. + - (-) Potential visual conflicts with the content +- (+) Indentation of the content is clear.
## Doc-comment examples -The following examples demonstrates the concrete usage scenarios: +The following examples demonstrate the concrete usage scenarios: `somefile.nix` @@ -246,7 +283,7 @@ The following examples demonstrates the concrete usage scenarios: ## Examples -This section contains examples for the different use-cases we would end up with; Visualize them and emphasize the previously discussed characteristics. +This section contains examples for the different use cases we would end up with; Visualize them and emphasize the previously discussed characteristics. ### Attribute bindings @@ -255,11 +292,11 @@ This section contains examples for the different use-cases we would end up with; ~~~nix { /** - mapAttrs is a well known function + mapAttrs is a well-known function # Examples - ```nix + "`Nix some code examples ``` @@ -270,16 +307,14 @@ This section contains examples for the different use-cases we would end up with; ### NixOS Module documentation -> This is an example of what is possible. The future conventions may still evolve from the nix-community - -NixOS Modules also can produce documentation from their interface declarations. However, this does not include a generic description and usage examples. +> Note: NixOS Modules also can produce documentation from their interface declarations. However, this does not include a generic description and usage examples. `myModule.nix` ~~~nix /** This is a custom module. - It configures a systemd service + It configures a systemd service. # Examples @@ -295,26 +330,24 @@ NixOS Modules also can produce documentation from their interface declarations. ### Anonymous function documentation -> This is an example of what is possible. The future conventions may still evolve from the nix-community - `function.nix` ~~~nix /** This is an anonymous function implementation. - It doesn't have a name yet. - But documentation can be right next to the implementation. + It does not have a name yet. + Nevertheless, documentation can be right next to the implementation. The name gets assigned later: - ```nix + "`Nix { plus = import ./function.nix; } ``` - (future) native nix or community tools provide - implementations to track doc-comments within the nix evaluator. + (Future) native Nix or community tools provide + implementations to track doc-comments within the Nix evaluator. Documentation of `sum` can then be inferred. This still needs to be specified/implemented! @@ -327,16 +360,14 @@ NixOS Modules also can produce documentation from their interface declarations. ### Anonymous expression documentation -> This is an example of what is possible. The future conventions may still evolve from the nix-community - `exp.nix` ~~~nix /** This is an anonymous string. - Its purpose can be documented. + It is the documentation for the "anonymous expression" use case. - Although this example is quite superficial, there might be use cases. + Although this example is relatively superficial, there might be use cases. */ "-p=libxml2/include/libxml2" ~~~ @@ -347,15 +378,15 @@ NixOS Modules also can produce documentation from their interface declarations. ## Changes the existing comments inside the code base. -This could mainly be automated. (e.g. via codemod) +This could be automated. (e.g., via codemod) Also, this affects only the `lib` folder and a few other places currently used to build the documentation. -## Changes in the `nixdoc`tooling is required. +## Changes in the `nixdoc' tooling are required. It remains an open question that needs to be tried; -Can we still build the current nixos manuals with the new standard? +Can we still build the current Nixos manuals with the new standard? - We can wait to change everything. - Migration can happen in small steps. @@ -385,7 +416,7 @@ Can we still build the current nixos manuals with the new standard? nixpkgs comments: -- `##` ~4k usages (most of them for visual separation e.g., `###########`) +- `##` ~4k usages (most of them for visual separation, e.g., `###########`) - `#` ~20k usages - `/*` ~6k usages - `/**` 160 usages (most empty ?) @@ -397,56 +428,54 @@ Choosing `/**` or subsets would cause minor conflicts within current nixpkgs. Wh While this allows the most freedom, it is usually considered the best option, not creating any restrictions. -But nix/rfc72 defines commonMark as the official documentation format. +However, nix/rfc72 defines commonMark as the official documentation format. This is why we decided to follow this convention. ## Consequences of not implementing this -- By not implementing this feature, nix gains no ability for tool-generated documentation. +- By not implementing this feature, Nix gains no ability for tool-generated documentation. - Documentation will be defined by nixdoc, not by the nix community. -- Many existing comments written for documentation will remain un-discoverable. +- Many existing comments written for documentation will remain imperceptible. # Unresolved questions [unresolved]: #unresolved-questions -- `nixodc` offers comments to describe function arguments, and this is currently discarded once they are directly described in prose by the user. - - Will `nix` itself implement native support like in rust -> `cargo doc`? -- How can a tool keep the connection from where a docstring was defined and where the attribute was exposed (lib/default. nix exposes mapAttrs which is defined at lib/attrsets.nix) +- How can a tool keep the connection from where a docstring was defined and where the attribute was exposed (lib/default. Nix exposes mapAttrs which is defined at lib/attrsets. Nix) - There are more complicated things. -**Answer**: This is a tooling question; implementation details won't be discussed in detail. It is possible to track comments in the AST. As shown before in this PR: [nix/#1652](https://github.com/NixOS/nix/pull/1652) +**Possible answer**: This is a tooling question; implementation details will not be discussed in detail. It is possible to track comments in the AST. As shown before in this PR: [nix/#1652](https://github.com/NixOS/nix/pull/1652) # Future work -[future]: #future-work +[Future]: #future-work ## Editor support - Implement displaying the related documentation when hovering over an expression. (lspAction/hover) -Nix already offers a bunch of LSP's e.g. [nil](https://github.com/oxalica/nil), [rnix-lsp](https://github.com/nix-community/rnix-lsp) are the most common ones. +Nix already offers a bunch of LSPs, e.g., [nil](https://github.com/oxalica/nil), [rnix-lsp](https://github.com/nix-community/rnix-lsp) are the most common ones. -## Nixodc +## Nixdoc Nixdoc must be changed to differentiate between regular comments and doc-comments. -There might be an intermediate phase of transition, where the old syntax and features are supported in parallel to allow a phase of transition and refactoring of existing documentation comments. +There might be an intermediate phase of transition, where the old Syntax and features are supported in parallel to allow a phase of transition and refactoring of existing documentation comments. ## Documentation generators - Future nixdoc could have a static `map.json` that contains the scope for every discovery file/path in nixpkgs. -As this second approach is much easier, We propose this is how we should initially start to extend the scope. +As this second approach is much easier, we propose that we initially start to extend the scope. -Generating documentation from doc-comments dynamically would still be excellent but remains a challenge. +Generating static documentation from doc-comments would still be excellent but remains a challenge if generated from Nix dynamically. If we want the fully automated approach, we need something to evaluate nix expressions. We solve such concerns in `tvix` or in `nix`, which could provide a `nix-analyzer` that pre-evaluates expressions and their respective documentation. ## Type -- An RFC under construction specifies the used syntax within the `# Type` Heading. +- An RFC under construction specifies the used Syntax within the `# Type` Heading. - The `type` feature should belong in the nix syntax. Try them within the comments first; This is still possible. - see a [preview](https://typednix.dev) of an eventual future doc-type-syntax. @@ -483,7 +512,7 @@ We solve such concerns in `tvix` or in `nix`, which could provide a `nix-analyze - [@davHau](https://github.com/davHau) - Author of [dream2nix](https://github.com/nix-community/dream2nix), (And many other nix-frameworks) -### About using comments as weak sources for typed nix. +### About using comments as weak sources for typed Nix. - [@roberth](https://github.com/roberth) - nixpkgs Architecture Team / nix core team ? - [@aakropotkin](https://github.com/aakropotkin/) - nixpkgs Architecture Team From 10976074d856ca1ace4ffcf934be5c3443d03acf Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Mon, 17 Apr 2023 09:44:44 +0200 Subject: [PATCH 049/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index c47a61974..d8d563a1f 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -500,19 +500,6 @@ We solve such concerns in `tvix` or in `nix`, which could provide a `nix-analyze ## Related discussions -### About doc-comments/doc-comments in general. - - [@documentation-team (meet @ 13.Apr 2023)](https://discourse.nixos.org/t/2023-04-13-documentation-team-meeting-notes-41/27264) -- [@flokli](https://github.com/flokli) - one of the [tvix](https://tvl.fyi/blog/rewriting-nix) authors -- [@tazjin](https://github.com/tazjin) - Original Author of `nixdoc`, one of the `tvix` authors - There is an actual but rather old PR (@roberth) that uses just comments to show documentation in the nix repl for functions. -> https://github.com/NixOS/nix/pull/1652 - -### About documentation approaches on independent frameworks. - -- [@davHau](https://github.com/davHau) - Author of [dream2nix](https://github.com/nix-community/dream2nix), (And many other nix-frameworks) - -### About using comments as weak sources for typed Nix. - -- [@roberth](https://github.com/roberth) - nixpkgs Architecture Team / nix core team ? -- [@aakropotkin](https://github.com/aakropotkin/) - nixpkgs Architecture Team From b570f1cb4fb362d2fceb7b4fa1b4b070bb5c390c Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Mon, 17 Apr 2023 09:53:29 +0200 Subject: [PATCH 050/110] Update 0145-doc-strings.md fix typo --- rfcs/0145-doc-strings.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index d8d563a1f..822295898 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -207,7 +207,7 @@ Each subsection here contains a decision along with arguments and counter-argume **Decision**: Doc-comments always relate to the expression in the next Abstract Syntax Tree Node (AST). -> Note: Precedence is unnecessary as the AST already decided upon that while parsing. In terms of precedence, a doc-comment virtually always has the lowest possible value (e.g., 1). However, this RFC will not define precedence. +> Note: Precedence is unnecessary as the AST already decided upon that while parsing. In terms of precedence, a doc-comment virtually always has the lowest possible value (e.g., -1). However, this RFC will not define precedence. > > Also Tool specific implementation questions must be resolved separately. Future RFCs to restrict or alter the behavior specified by this decision are valid if they present rational reasoning. @@ -227,7 +227,7 @@ Each subsection here contains a decision along with arguments and counter-argume -## Single doc-comments (do not exist) +## Single-line doc-comments (do not exist) **Observing**: Nix offers two variants of comments; single- and multi-line comments. There may be use cases where it is desirable to have a form of single-line comments subtyped for doc-comment purposes. @@ -235,8 +235,6 @@ Each subsection here contains a decision along with arguments and counter-argume **Decision**: Single-line comments **cannot be used** in any form for documentation puposes. -> This decision was not easy and may still be controversial -
Arguments From 48477b1ba3e841046b2a38a12f8902444f8802ca Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Mon, 17 Apr 2023 09:56:15 +0200 Subject: [PATCH 051/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 822295898..ff101e8ba 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -229,7 +229,7 @@ Each subsection here contains a decision along with arguments and counter-argume ## Single-line doc-comments (do not exist) -**Observing**: Nix offers two variants of comments; single- and multi-line comments. There may be use cases where it is desirable to have a form of single-line comments subtyped for doc-comment purposes. +**Observing**: Nix offers two variants of comments; single- (`#`) and multi-line comments (`/* */`). There may be use cases where it is desirable to have a form of single-line comments subtyped for doc-comment purposes. **Considering**: Single-line comment for documentation. From 3c1a8f77f1c5e88451bf1f90e69a6f8cdee8ac0c Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 20 Apr 2023 12:08:16 +0200 Subject: [PATCH 052/110] clearify: doc comment reference logic Thanks @tajin --- rfcs/0145-doc-strings.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index ff101e8ba..f72e1cca1 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -205,11 +205,9 @@ Each subsection here contains a decision along with arguments and counter-argume **Considering**: General reference logic between doc-comments and expressions. -**Decision**: Doc-comments always relate to the expression in the next Abstract Syntax Tree Node (AST). +**Decision**: Doc-comments refer to the **immediately** preceded expression. Only whitespaces are allowed in between. -> Note: Precedence is unnecessary as the AST already decided upon that while parsing. In terms of precedence, a doc-comment virtually always has the lowest possible value (e.g., -1). However, this RFC will not define precedence. -> -> Also Tool specific implementation questions must be resolved separately. Future RFCs to restrict or alter the behavior specified by this decision are valid if they present rational reasoning. +> Note: We are working on a generic example implementation that covers about 90% of all documentation use cases.
Arguments From 834f8d4a784f0830f9586d02b8e767a75060730a Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 20 Apr 2023 21:06:33 +0200 Subject: [PATCH 053/110] Update rfcs/0145-doc-strings.md Co-authored-by: Anderson Torres --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index f72e1cca1..f9d52a788 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -38,7 +38,7 @@ It allows distinguishing between regular and doc comments. Having distinction is ## Current State -Currently, Nix does not specify doc comments. Over time there evolved certain conventions, But the nix-community never specified those. +Currently, Nix does not have doc comments. Over time there evolved certain conventions, but they were never formally specified. Within this RFC, we want to clarify which convention we should use, harmonize and improve the format for doc-comments to achieve the envisioned goals. We use a tool called [nixdoc](https://github.com/nix-community/nixdoc) to use a `doc-comment'- like functionality, which allows us to build a subset of static documentation of the `nixpkgs.lib` functions. From e693a28eb18bbd16b21d8f9726850a9be844b1db Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sat, 29 Apr 2023 12:02:30 +0200 Subject: [PATCH 054/110] Update rfcs/0145-doc-strings.md Co-authored-by: Florian Klink --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index f9d52a788..1b6fb2343 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -69,7 +69,7 @@ Here is an example of the format understood by *nixdoc*: ## Current problems -### Lack of having an agreed Format +### Lack of having an agreed format The Nix community interprets the outer format and content of doc-comments differently. Within nixpkgs, several conventions and patterns still need to be harmonized. Some are compatible with nixdoc, while others do not and decide upon their conventions. We immediately observe one clear thing: Most of them utilize some form of markdown, even if the content is not rendered. From fc929412a88b312acedd2f4e169a1949fdcbf2b1 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sat, 29 Apr 2023 12:02:43 +0200 Subject: [PATCH 055/110] Update rfcs/0145-doc-strings.md Co-authored-by: Florian Klink --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 1b6fb2343..6bb791511 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -41,7 +41,7 @@ It allows distinguishing between regular and doc comments. Having distinction is Currently, Nix does not have doc comments. Over time there evolved certain conventions, but they were never formally specified. Within this RFC, we want to clarify which convention we should use, harmonize and improve the format for doc-comments to achieve the envisioned goals. -We use a tool called [nixdoc](https://github.com/nix-community/nixdoc) to use a `doc-comment'- like functionality, which allows us to build a subset of static documentation of the `nixpkgs.lib` functions. +We use a tool called [nixdoc](https://github.com/nix-community/nixdoc) to use a 'doc-comment'- like functionality, which allows us to build a subset of static documentation of the `nixpkgs.lib` functions. The format of a doc-comment was, however, never specified. The nix community does not understand the rules enforced by `nixdoc`, which we believe is one of the reasons for its lack of adoption. From c30cb3331c15c4e6980150c4a5a07e1fc246eeac Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sat, 29 Apr 2023 12:03:48 +0200 Subject: [PATCH 056/110] Update rfcs/0145-doc-strings.md Co-authored-by: Florian Klink --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 6bb791511..1f87ba155 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -75,7 +75,7 @@ The Nix community interprets the outer format and content of doc-comments differ In general, the format for writing documentation strings is **not specified**. -The *nixdoc*-tool enforces a somewhat consistent format, but the basic format was never specified and cannot be enforced in parts of nixpkgs where nixdoc is currently not applied. +The *nixdoc*-tool requires a somewhat consistent format, but the basic format was never specified and cannot be enforced in parts of nixpkgs where nixdoc is currently not applied. Extending the scope of *nixdoc* is not the primary goal. Instead, we should find formal rules for writing *doc-comments*. Tools like *nixdoc* can then implement against this RFC instead of the format relying on nixdoc implementation details. From bf9cf3c5b9f483b4b8804dce1bfdabadb2c2f2cb Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 30 Apr 2023 15:43:42 +0200 Subject: [PATCH 057/110] Update rfcs/0145-doc-strings.md Co-authored-by: Mykola Orliuk --- rfcs/0145-doc-strings.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 1f87ba155..b839074e3 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -26,10 +26,9 @@ However, both concerns relate closely to each other; It makes sense and reduces The following are the envisioned goals. - Be able to generate documentation from code for any nix expression. -- Be able to distinguish between documentation-relevant comments and unrelated comments. - Make doc comments easy to write and read -- Be able to parse and render doc comments nicely - Standardize a format for doc comments that further RFCs can extend + - E.g. extend with code examples that can be converted to tests (aka doctests). This RFC is a significant change to existing documentation conventions. It allows distinguishing between regular and doc comments. Having distinction is essential because arbitrary code comments should not end up in generated documentation. From caa849abb760e25becad5d9cbbb8c96b8c11a0b3 Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Sun, 30 Apr 2023 17:25:17 +0200 Subject: [PATCH 058/110] clarify & add some more examples --- rfcs/0145-doc-strings.md | 205 ++++++++++++++++++++++++++------------- 1 file changed, 139 insertions(+), 66 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 1f87ba155..1c5a7e10d 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -71,7 +71,7 @@ Here is an example of the format understood by *nixdoc*: ### Lack of having an agreed format -The Nix community interprets the outer format and content of doc-comments differently. Within nixpkgs, several conventions and patterns still need to be harmonized. Some are compatible with nixdoc, while others do not and decide upon their conventions. We immediately observe one clear thing: Most of them utilize some form of markdown, even if the content is not rendered. +The Nix community interprets the outer format and content of doc-comments differently. Within nixpkgs, several conventions and patterns need to be harmonized. It seems every directory decides upon their own convention; Or doesn't follow any. We immediately observe one clear thing: Most of comments utilize some form of markdown, even if they are not rendered. In general, the format for writing documentation strings is **not specified**. @@ -123,8 +123,7 @@ Each subsection here contains a decision along with arguments and counter-argume ## Discontinue nixdoc's custom format -**Observing**: Nixdoc currently provides some format that can generate documentation from a small fraction of files in nixpkgs, Namely the files for `.lib` functions. The folder provides a consistent community convention that allows the building of static documentation. Applying nixdoc to files outside that directory is only partially possible due to the lack of a standard doc-comment format. - +**Observing**: Nixdoc currently provides some format that can generate documentation from a small fraction of files in nixpkgs, Namely the files for `.lib` functions. The folder provides a consistent community convention that allows the building of static documentation. Applying nixdoc to files outside that directory is only partially possible due to the lack of a standard doc-comment format. **Considering**: Should we propose the `nixdoc` format as a community standard? @@ -134,16 +133,16 @@ Each subsection here contains a decision along with arguments and counter-argume Arguments - (+) Stays primarily compatible with the currently used comments in lib. - - (-) Outside the ./lib folder, nixodc is not a convention. + - (-) Outside the ./lib folder, nixodc is not a convention. - (-) Is not widely understood. -- (-) Is not widely used. (e.g., Outside the /lib folder) - (-) The format is specific, should be more generic, and follow established conventions. (e.g., Markdown, Doxygen, JSdoc, Haddock) - (-) The format is too complex and needs a whole list of complex rules to describe the exact format and content rules; this can also be observed in many false-formatted comments that do not render correctly. - (-) The sections are implicitly continued. - (-) The predefined section headers/keywords are very close to natural language and may accidentally be used. - (-) Syntax to start new sections does not follow conventions (e.g., `Example:`) In contrast JSdoc defines `@Example`; Markdown `# Example`; Doxygen `@example`; Haskell-Haddock `>>>` - (-) The content is implicitly rendered differently depending on predefined rules. Those rules need to be understood/feel natural. -- (+) It Is already quite established and has some adoptions outside the `/lib` folder. +- (+) Is already quite established in /lib. +- (+) Some usages outside of /lib indicate the process of adoption. - (-) Some multiline comments become doc-comments, but it is unclear which comments have such a property.
@@ -156,6 +155,29 @@ Each subsection here contains a decision along with arguments and counter-argume **Decision**: use `/** {content} */` as the outer format. +`Example` + +````nix +{ + /** + The identity function + + Describes in every detail why + this function is important and how to use it. + + # Examples + + ``` + id "foo" + -> + "foo" + ``` + + */ + id = x: x; +} +```` +
Arguments @@ -163,6 +185,7 @@ Each subsection here contains a decision along with arguments and counter-argume - (+) Is a strict subset of multiline comments, which allows multiline documentation natively. - (+) Does not need Language Server (LSP) support for productive usage. (In contrast to, e.g., `##`) - (+) Allows copy-pasting content without the need for re-formatting. + - (-) Partially re-formatting just like in multiline Nix strings (`''`) might still be needed. - (-) Is visually less distinctive. - (-) Indentation is more complex and visually less present. - (+) It would be most intuitive if its indentation logic implements Nix's magic multiline strings. @@ -195,11 +218,11 @@ Each subsection here contains a decision along with arguments and counter-argume - (-) Strictly binding doc comments content to commonMark might restrict users. - (+) Users/Tools can still use regular comments or develop alternative solutions. - (-) CommonMark does not specify the current rich features from nixdoc, such as predefined sections and structures that could be used as a source for automated toolings. Such as types for type-checking or examples to run automated-tests - - (+) Future tools can still build their conventions on top of this RFC. They might not directly specify them in an RFC but be a tool developers choose for a specific codebase. However, we have yet to get that tools. So it is good when rich features remain unspecified. + - (+) Future tools can still build their conventions on top of this RFC. They might not directly specify them in an RFC but be a tool developers choose for a specific codebase. However, we have yet to get that tools. So it is good when rich features remain unspecified.
-## Place doc-comments before the referenced expression +## Place doc-comments before the referenced expression **Observing**: Doc-comments currently are placed above the expression they document. More precisely, only named attribute bindings `foo = ` can be documented. There is also the need to support documentation for anonymous functions. More generally, there is the need to document anonymous expressions where they are defined. @@ -227,11 +250,11 @@ Each subsection here contains a decision along with arguments and counter-argume ## Single-line doc-comments (do not exist) -**Observing**: Nix offers two variants of comments; single- (`#`) and multi-line comments (`/* */`). There may be use cases where it is desirable to have a form of single-line comments subtyped for doc-comment purposes. +**Observing**: Nix offers two variants of comments; single- (`#`) and multi-line comments (`/* */`). There may be use cases where it is desirable to have a form of single-line comments subtyped for doc-comment purposes. -**Considering**: Single-line comment for documentation. +**Considering**: Single-line comment for documentation. (Starting a doc-comment with `#`) -**Decision**: Single-line comments **cannot be used** in any form for documentation puposes. +**Decision**: Single-line comments (starting with `#`) **cannot be used** in any form for documentation puposes.
Arguments @@ -251,11 +274,11 @@ Each subsection here contains a decision along with arguments and counter-argume ## Doc-comment examples -The following examples demonstrate the concrete usage scenarios: +This section contains examples for the different use cases we would end up with; Visualize them and emphasize the previously discussed characteristics. `somefile.nix` -~~~nix +````nix { /** Documentation for mapAttrs @@ -275,38 +298,90 @@ The following examples demonstrate the concrete usage scenarios: */ mapAttrs = f: s: <...>; } -~~~ +```` -## Examples +### Attribute bindings -This section contains examples for the different use cases we would end up with; Visualize them and emphasize the previously discussed characteristics. +`somefile.nix` -### Attribute bindings +````nix + { + /** -`somefile.nix` + mapAttrs is a well-known function -~~~nix -{ - /** - mapAttrs is a well-known function - - # Examples - - "`Nix - some code examples - ``` - - */ - mapAttrs = f: s: #... -} -~~~ + # Examples + + ```Nix + some code examples + ``` + + */ + mapAttrs = f: s: #... + } +```` -### NixOS Module documentation +### Indentation follows know nix's `''` behavior + +````nix + { + /** + Line 1 + Line 2 + */ + id = f: s: #... + } +```` +-> +```markdown +Line 1 + Line 2 +``` + +Or with commonmark + +````nix + { + /** + # Title + + Some foo bar + + ## Subtitle + + Some more + + indented 1 + + indented 2 + + */ + id = f: s: #... + } +```` +-> +```markdown +# Title + +Some foo bar + +## Subtitle + +Some more + + indented 1 + + indented 2 + +``` + + +### NixOS Module documentation > Note: NixOS Modules also can produce documentation from their interface declarations. However, this does not include a generic description and usage examples. `myModule.nix` -~~~nix +````nix /** This is a custom module. @@ -316,18 +391,17 @@ This section contains examples for the different use cases we would end up with; Different use case scenarios how to use this module - */ {config, ...}: { config = f: s: f s; } -~~~ +```` -### Anonymous function documentation +### Anonymous function documentation `function.nix` -~~~nix +````nix /** This is an anonymous function implementation. @@ -336,7 +410,7 @@ This section contains examples for the different use cases we would end up with; The name gets assigned later: - "`Nix + ```nix { plus = import ./function.nix; } @@ -352,12 +426,12 @@ This section contains examples for the different use cases we would end up with; { sum = a + b; } -~~~ +```` -### Anonymous expression documentation +### Anonymous expression documentation `exp.nix` -~~~nix +````nix /** This is an anonymous string. @@ -366,19 +440,18 @@ This section contains examples for the different use cases we would end up with; Although this example is relatively superficial, there might be use cases. */ "-p=libxml2/include/libxml2" -~~~ +```` # Drawbacks [drawbacks]: #drawbacks - -## Changes the existing comments inside the code base. +## Changes the existing comments inside the code base This could be automated. (e.g., via codemod) Also, this affects only the `lib` folder and a few other places currently used to build the documentation. -## Changes in the `nixdoc' tooling are required. +## Changes in the `nixdoc` tooling are required It remains an open question that needs to be tried; @@ -397,14 +470,14 @@ Can we still build the current Nixos manuals with the new standard? | Inspired by | Rust | Current nixpkgs.lib | C++/Java/Javascript | Haskell Haddock | | Changes the existing code by | Much | Less | Even More | Less | | Needs Termination | No | Yes | Yes | Yes | -| Indentation | Clear | like Nix's multiline strings, thus **Intuitive** | Poor | ? | -| Needs vertical space | No | Yes | Yes | Yes | +| Indentation | Clear | like Nix's multiline strings, thus **Intuitive** | Clear | ? | +| Needs vertical space | No | Yes | Yes | Yes | | Visual distinction from comments | High | Low | Medium | Medium | | Needs Autocompletion (Language Server) to continue the next line. | Yes | No | Yes | No | | Punctuation Variations / Amount of different special characters | 1 (Less) | 2 (Medium) | 2 (Medium) | 3 (More) | -| Markdown compatibility (also depends on indentation clarity) | Good, but visual conflicts with headings` #` | Good | Medium | Good | +| Markdown compatibility (also depends on indentation clarity) | Good, but visual conflicts with headings `# Title` | Good | Medium | Good | | breaks when interrupted with newlines | Yes | No | ? | No | -| Simplicity (Brainload) | Medium | Simple | Complex | More Complex | +| Simplicity (Brainload) | Medium | Simple | Complex | More Complex | ### Refactoring note @@ -416,15 +489,14 @@ nixpkgs comments: - `#` ~20k usages - `/*` ~6k usages - `/**` 160 usages (most empty ?) - -Choosing `/**` or subsets would cause minor conflicts within current nixpkgs. While this is NOT the main reason for the final decision, it MUST be considered. +Choosing `/**` or subsets would cause minor conflicts within current nixpkgs. While this is NOT the main reason for the final decision, it MUST be considered. ## Just free text as a content format -While this allows the most freedom, it is usually considered the best option, not creating any restrictions. +While this allows the most freedom, it is usually considered the best option, not creating any restrictions. -However, nix/rfc72 defines commonMark as the official documentation format. +However, nix/rfc72 defines commonMark as the official documentation format. This is why we decided to follow this convention. ## Consequences of not implementing this @@ -452,22 +524,23 @@ This is why we decided to follow this convention. Nix already offers a bunch of LSPs, e.g., [nil](https://github.com/oxalica/nil), [rnix-lsp](https://github.com/nix-community/rnix-lsp) are the most common ones. - ## Nixdoc -Nixdoc must be changed to differentiate between regular comments and doc-comments. +Nixdoc may be changed to differentiate between regular comments and doc-comments. There might be an intermediate phase of transition, where the old Syntax and features are supported in parallel to allow a phase of transition and refactoring of existing documentation comments. ## Documentation generators -- Future nixdoc could have a static `map.json` that contains the scope for every discovery file/path in nixpkgs. +We think that a future documentation tool could be out of one of the two following categories. + +- (1) Tools that utilize static code analysis and configuration files. -As this second approach is much easier, we propose that we initially start to extend the scope. +- (2) Tools that use dynamic evaluation to solve for tracking down name and value relationships and provide more accurate documentation with less configuration overhead. -Generating static documentation from doc-comments would still be excellent but remains a challenge if generated from Nix dynamically. -If we want the fully automated approach, we need something to evaluate nix expressions. + > We could solve such concerns in [`tvix`](https://tvix.dev/) or in `nix`, which could vend a tool that pre-evaluates expressions and gathers their respective documentation. -We solve such concerns in `tvix` or in `nix`, which could provide a `nix-analyzer` that pre-evaluates expressions and their respective documentation. +For the beginning it seems promising to start with the static approach (1). In the long term dynamic approach (2) seems more promising and accurate but requires a much deeper understanding of compilers and evaluators. +However the border between those two variants is not strict and the community might even build tools that live in between. ## Type @@ -478,17 +551,17 @@ We solve such concerns in `tvix` or in `nix`, which could provide a `nix-analyze ## Native support in Nix -- `NixOS/nix` should implement native support for doc-comments so that our users do not have to rely on nixpkgs or external tools. Those tools can still exist and provide more custom functionality, but documenting nix expressions should be natively possible. +- `NixOS/nix` could implement native support for doc-comments so that our users do not have to rely on nixpkgs or external tools. Those tools can still exist and provide more custom functionality, but documenting nix expressions should be natively possible. -# References +## References -## Other Conventions +### Other Conventions - [Rust](https://doc.rust-lang.org/stable/reference/comments.html#doc-comments) - [Python](https://peps.python.org/pep-0257/) - [JSDoc](https://jsdoc.app/) -## Related tools +### Related tools - [Nixdoc](https://github.com/nix-community/nixdoc) - [Rustdoc](https://doc.rust-lang.org/rustdoc/how-to-write-documentation.html) From 6790f9dadbe36197a2deecd8dacf7ae2801afb2a Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 30 Apr 2023 17:50:39 +0200 Subject: [PATCH 059/110] Update rfcs/0145-doc-strings.md Co-authored-by: asymmetric --- rfcs/0145-doc-strings.md | 1 - 1 file changed, 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 35bf659be..24142bca2 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -96,7 +96,6 @@ Having a distinction would allow us to - [trivial-builders](https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/trivial-builders.nix) - [stdenv/mkDerivation](https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix) - [nixos/lib/make-disk-image](https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-disk-image.nix) -- more #### frameworks From 298955fa71b9c19b870168304bf316f75cde16cb Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 30 Apr 2023 18:17:32 +0200 Subject: [PATCH 060/110] Update rfcs/0145-doc-strings.md Co-authored-by: asymmetric --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 24142bca2..e3c399224 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -57,7 +57,7 @@ Here is an example of the format understood by *nixdoc*: */ AttrFunc = - # + # arg1: # arg2: From 477c88050e70cdea2780bc76eab229eda4a6dbec Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 30 Apr 2023 18:18:40 +0200 Subject: [PATCH 061/110] Update rfcs/0145-doc-strings.md Co-authored-by: asymmetric --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index e3c399224..dee9d2486 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -72,7 +72,7 @@ Here is an example of the format understood by *nixdoc*: The Nix community interprets the outer format and content of doc-comments differently. Within nixpkgs, several conventions and patterns need to be harmonized. It seems every directory decides upon their own convention; Or doesn't follow any. We immediately observe one clear thing: Most of comments utilize some form of markdown, even if they are not rendered. -In general, the format for writing documentation strings is **not specified**. +In general, the format for writing documentation strings is **not formally specified**. The *nixdoc*-tool requires a somewhat consistent format, but the basic format was never specified and cannot be enforced in parts of nixpkgs where nixdoc is currently not applied. From 6d4a327bd3437e33c3a5b66053acd96bccf8a80e Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 30 Apr 2023 18:56:57 +0200 Subject: [PATCH 062/110] Update rfcs/0145-doc-strings.md Co-authored-by: asymmetric --- rfcs/0145-doc-strings.md | 1 - 1 file changed, 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index dee9d2486..8a7c719e0 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -101,7 +101,6 @@ Having a distinction would allow us to - [dream2nix/utils](https://github.com/nix-community/dream2nix/blob/main/src/utils/config.nix) - [dream2nix/templates/builder](https://github.com/nix-community/dream2nix/blob/main/src/templates/builders/default.nix) -- more #### Current tools From e09421019e9e82858212e2046161fe0014e33c4c Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 30 Apr 2023 19:36:19 +0200 Subject: [PATCH 063/110] update nixdoc format according to current nixodc-maintainer Co-authored-by: asymmetric --- rfcs/0145-doc-strings.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 8a7c719e0..61dde6a8a 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -50,13 +50,15 @@ Here is an example of the format understood by *nixdoc*: # nixpkgs/lib/attrsets.nix { - /* - Example: - + /* + Type: + + Example: + */ - AttrFunc = + Attr = # arg1: # From 05c3c9d89afe78a3b92414f6da4088af133eb8c2 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 30 Apr 2023 19:37:42 +0200 Subject: [PATCH 064/110] Update rfcs/0145-doc-strings.md Co-authored-by: Florian Klink --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 61dde6a8a..d1def8d64 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -148,7 +148,7 @@ Each subsection here contains a decision along with arguments and counter-argume ## `/**` to start a doc-comment -**Observing**: Doc-comments' outer format should be a distinctive subset of regular comments. Nevertheless, it should allow native writing without an ide or full-blown-language servers. +**Observing**: Doc-comments' outer format should be a distinctive subset of regular comments. Nevertheless, it should allow native writing without an IDE or editor support. **Considering**: `/** {content} */` where `{content}` is the inner format which is discussed later. From 6ad84801d2a76299ad24d156403b8a54ebf02e18 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 30 Apr 2023 19:46:08 +0200 Subject: [PATCH 065/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index d1def8d64..9334b4fa6 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -92,6 +92,10 @@ Having a distinction would allow us to ### References to the problems above +> Note: That specific collection of links should show the amout of inconsistency within the Nix ecosystem, which is a target of this RFC. +> +> The "Current tools" section contains links to some more creative tooling solutions that eveloped, while still not providing the user experience that developers are used from other ecosystems. + #### nixpkgs - Dosctrings examples - [lib/attrsets](https://github.com/NixOS/nixpkgs/blob/master/lib/attrsets.nix) @@ -182,7 +186,7 @@ Each subsection here contains a decision along with arguments and counter-argume - (+) It is mostly compatible with the currently used multiline comments. - (+) Is a strict subset of multiline comments, which allows multiline documentation natively. -- (+) Does not need Language Server (LSP) support for productive usage. (In contrast to, e.g., `##`) +- (+) Does not need editor support for productive usage. (In contrast to, e.g., `##`) - (+) Allows copy-pasting content without the need for re-formatting. - (-) Partially re-formatting just like in multiline Nix strings (`''`) might still be needed. - (-) Is visually less distinctive. @@ -227,7 +231,7 @@ Each subsection here contains a decision along with arguments and counter-argume **Considering**: General reference logic between doc-comments and expressions. -**Decision**: Doc-comments refer to the **immediately** preceded expression. Only whitespaces are allowed in between. +**Decision**: Doc-comment refer to the expression that is **immediately** next to it. Only whitespaces are allowed between doc-comment and following expression. > Note: We are working on a generic example implementation that covers about 90% of all documentation use cases. From 321fbce0a687bc5daf80c0336d53f2502e203d63 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 30 Apr 2023 19:48:28 +0200 Subject: [PATCH 066/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 9334b4fa6..8bd674682 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -84,11 +84,7 @@ Extending the scope of *nixdoc* is not the primary goal. Instead, we should find The format does not allow any distinction between doc comments and regular comments. -Having a distinction would allow us to - -1. Find all comments that are part of the documentation -2. Render them in the documentation format -3. Connect the documentation to the exact places in the Nix code. +Having a distinction would allow us to build reliable and correct documentation ### References to the problems above From 3066a3e333a8a92fcaf9d89b5a8c0c3f5f4f8c60 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 30 Apr 2023 19:49:57 +0200 Subject: [PATCH 067/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 8bd674682..5036d3b18 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -42,7 +42,7 @@ Within this RFC, we want to clarify which convention we should use, harmonize an We use a tool called [nixdoc](https://github.com/nix-community/nixdoc) to use a 'doc-comment'- like functionality, which allows us to build a subset of static documentation of the `nixpkgs.lib` functions. -The format of a doc-comment was, however, never specified. The nix community does not understand the rules enforced by `nixdoc`, which we believe is one of the reasons for its lack of adoption. +The format of a doc-comment was, however, never specified. The nix community does not understand the rules required by `nixdoc`, which we believe is one of the reasons for its lack of adoption. Here is an example of the format understood by *nixdoc*: From 92919d7e49193d99f7a0585422dbe023dea732b5 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 30 Apr 2023 19:54:30 +0200 Subject: [PATCH 068/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 5036d3b18..c15d812c2 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -90,7 +90,7 @@ Having a distinction would allow us to build reliable and correct documentation > Note: That specific collection of links should show the amout of inconsistency within the Nix ecosystem, which is a target of this RFC. > -> The "Current tools" section contains links to some more creative tooling solutions that eveloped, while still not providing the user experience that developers are used from other ecosystems. +> The "Current tools" section contains links to more creative tooling solutions that developed while still not providing the user experience that developers are used to from other ecosystems. #### nixpkgs - Dosctrings examples From 36adf876cdb01764f60df7b8f4fdeda4d7f2473c Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 30 Apr 2023 20:05:39 +0200 Subject: [PATCH 069/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index c15d812c2..714a387f0 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -122,7 +122,7 @@ Each subsection here contains a decision along with arguments and counter-argume ## Discontinue nixdoc's custom format -**Observing**: Nixdoc currently provides some format that can generate documentation from a small fraction of files in nixpkgs, Namely the files for `.lib` functions. The folder provides a consistent community convention that allows the building of static documentation. Applying nixdoc to files outside that directory is only partially possible due to the lack of a standard doc-comment format. +**Observing**: Nixdoc currently provides some format that can generate documentation from a small fraction of files in nixpkgs, Namely the files for `.lib` functions. The directory provides a consistent community convention that allows the building of static documentation. Applying nixdoc to files outside that directory is only partially possible due to the lack of a standard doc-comment format. **Considering**: Should we propose the `nixdoc` format as a community standard? @@ -132,7 +132,7 @@ Each subsection here contains a decision along with arguments and counter-argume Arguments - (+) Stays primarily compatible with the currently used comments in lib. - - (-) Outside the ./lib folder, nixodc is not a convention. + - (-) Outside the ./lib directory, nixodc is not a convention. - (-) Is not widely understood. - (-) The format is specific, should be more generic, and follow established conventions. (e.g., Markdown, Doxygen, JSdoc, Haddock) - (-) The format is too complex and needs a whole list of complex rules to describe the exact format and content rules; this can also be observed in many false-formatted comments that do not render correctly. @@ -140,8 +140,7 @@ Each subsection here contains a decision along with arguments and counter-argume - (-) The predefined section headers/keywords are very close to natural language and may accidentally be used. - (-) Syntax to start new sections does not follow conventions (e.g., `Example:`) In contrast JSdoc defines `@Example`; Markdown `# Example`; Doxygen `@example`; Haskell-Haddock `>>>` - (-) The content is implicitly rendered differently depending on predefined rules. Those rules need to be understood/feel natural. -- (+) Is already quite established in /lib. -- (+) Some usages outside of /lib indicate the process of adoption. +- (+) Some usages outside of /lib directory indicate the process of adoption. - (-) Some multiline comments become doc-comments, but it is unclear which comments have such a property.
@@ -448,7 +447,7 @@ Some more This could be automated. (e.g., via codemod) -Also, this affects only the `lib` folder and a few other places currently used to build the documentation. +Also, this affects only the `lib` directory and a few other places currently used to build the documentation. ## Changes in the `nixdoc` tooling are required From cd3cf3377842fc06166d37969cb6fb75f70eb881 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Mon, 1 May 2023 17:47:53 +0200 Subject: [PATCH 070/110] Update rfcs/0145-doc-strings.md Co-authored-by: asymmetric --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 714a387f0..45299efa0 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -494,7 +494,7 @@ Choosing `/**` or subsets would cause minor conflicts within current nixpkgs. Wh While this allows the most freedom, it is usually considered the best option, not creating any restrictions. -However, nix/rfc72 defines commonMark as the official documentation format. +However, [RFC72](https://github.com/NixOS/rfcs/blob/master/rfcs/0072-commonmark-docs.md) defines commonMark as the official documentation format. This is why we decided to follow this convention. ## Consequences of not implementing this From 65d33a6152d69b494ae8c84cdbf51994b12119a6 Mon Sep 17 00:00:00 2001 From: hsjobeki Date: Sat, 6 May 2023 17:43:01 +0200 Subject: [PATCH 071/110] work feedback from @asymmetric in --- rfcs/0145-doc-strings.md | 96 ++++++++++++++++++---------------------- 1 file changed, 43 insertions(+), 53 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 45299efa0..d109d605d 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -16,7 +16,7 @@ Propose a standard format specification for Doc-comments. This RFC includes two concerns that define a doc-comment: - Outer format rules to allow distinction between regular comments and doc-comments -- Inner format rules that describe the required format of the content. +- Inner format rules that describe the required format of the content. However, both concerns relate closely to each other; It makes sense and reduces bureaucracy to address that in a single RFC. @@ -33,7 +33,12 @@ The following are the envisioned goals. This RFC is a significant change to existing documentation conventions. It allows distinguishing between regular and doc comments. Having distinction is essential because arbitrary code comments should not end up in generated documentation. -> Hint: Generating static documentation is controvert topic in nixpkgs. We found that it is impossible to generate accurate documentation statically. A correct solution would involve the evaluation of expressions in some way. +> Note: Generating static documentation is a controversial topic in nixpkgs.[^1][^2][^3][^4] We found that it is very hard to generate accurate documentation statically. A correct solution would involve the evaluation of expressions in some way. + +[^1]: [Unmerged PR: nix repl doc-comments](https://github.com/NixOS/nix/pull/1652), 2017 +[^2]: [Unmerged PR: doc-strings for functors](https://github.com/NixOS/nix/pull/5527), 2021 +[^3]: [discourse: users asking how to obtain documentation](https://discourse.nixos.org/t/how-to-generate-documentation-from-arbitrary-nix-code/22292), 2022 +[^4]: [discourse: aksing for consensus on documentation](https://discourse.nixos.org/t/any-consensus-on-documentation-generation-tool-for-nixpkgs-manual/15550), 2021 ## Current State @@ -88,26 +93,19 @@ Having a distinction would allow us to build reliable and correct documentation ### References to the problems above -> Note: That specific collection of links should show the amout of inconsistency within the Nix ecosystem, which is a target of this RFC. -> -> The "Current tools" section contains links to more creative tooling solutions that developed while still not providing the user experience that developers are used to from other ecosystems. +> Note: That specific collection of links should show the amount of inconsistency within the Nix ecosystem, which is a target of this RFC. -#### nixpkgs - Dosctrings examples +#### nixpkgs - doc-comment examples - [lib/attrsets](https://github.com/NixOS/nixpkgs/blob/master/lib/attrsets.nix) - [trivial-builders](https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/trivial-builders.nix) - [stdenv/mkDerivation](https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix) - [nixos/lib/make-disk-image](https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-disk-image.nix) -#### frameworks - -- [dream2nix/utils](https://github.com/nix-community/dream2nix/blob/main/src/utils/config.nix) -- [dream2nix/templates/builder](https://github.com/nix-community/dream2nix/blob/main/src/templates/builders/default.nix) - #### Current tools -Other tools that work directly with the nix AST and comments. -Those would strategically benefit from this RFC, even if it means refactoring and migration efforts. +We collected some Tools that work directly with the nix AST and comments. +Those would benefit from a standardized doc-comment. - [noogle](https://noogle.dev) - Nix API search engine. It allows searching functions and other expressions. - [nix-doc](https://github.com/lf-/nix-doc) - A Nix developer tool leveraging the rnix Nix parser for intelligent documentation search and tags generation @@ -272,29 +270,31 @@ Each subsection here contains a decision along with arguments and counter-argume ## Doc-comment examples -This section contains examples for the different use cases we would end up with; Visualize them and emphasize the previously discussed characteristics. +This section contains many examples for some different use cases we; Visualize them and emphasize the previously discussed characteristics. `somefile.nix` ````nix { /** - Documentation for mapAttrs + Documentation for the fundamental 'id' function - # Example + # Examples ``` - mapAttrs {attr = "foo"; } ... + id 1 + => + 1 ``` # Type ``` - mapAttrs :: a -> b -> c + id :: a -> a ``` */ - mapAttrs = f: s: <...>; + id = x: x; } ```` @@ -310,8 +310,8 @@ This section contains examples for the different use cases we would end up with; # Examples - ```Nix - some code examples + ``` + # some code examples ``` */ @@ -321,6 +321,9 @@ This section contains examples for the different use cases we would end up with; ### Indentation follows know nix's `''` behavior +Indentation follows the know `''`-nix multiline strings behavior. +Making usage more intuitive and and one point less to think about. + ````nix { /** @@ -336,7 +339,7 @@ Line 1 Line 2 ``` -Or with commonmark +Or more advanced ````nix { @@ -357,7 +360,7 @@ Or with commonmark id = f: s: #... } ```` --> +=> ```markdown # Title @@ -406,19 +409,7 @@ Some more It does not have a name yet. Nevertheless, documentation can be right next to the implementation. - The name gets assigned later: - - ```nix - { - plus = import ./function.nix; - } - ``` - - (Future) native Nix or community tools provide - implementations to track doc-comments within the Nix evaluator. - Documentation of `sum` can then be inferred. - This still needs to be specified/implemented! - + The name gets assigned later. */ {a, b}: { @@ -462,7 +453,7 @@ Can we still build the current Nixos manuals with the new standard? [alternatives]: #alternatives ## All considered outer formats - + | Property / Approach | `##` | `/** */` | `Javadoc` | `/*\|` or `/*^` | |---|---|---|---|---| | Inspired by | Rust | Current nixpkgs.lib | C++/Java/Javascript | Haskell Haddock | @@ -476,11 +467,11 @@ Can we still build the current Nixos manuals with the new standard? | Markdown compatibility (also depends on indentation clarity) | Good, but visual conflicts with headings `# Title` | Good | Medium | Good | | breaks when interrupted with newlines | Yes | No | ? | No | | Simplicity (Brainload) | Medium | Simple | Complex | More Complex | - + ### Refactoring note - + **Observing**: From a refactoring perspective, it might also be interesting to see how many conflicts the different formats would cause. - + nixpkgs comments: - `##` ~4k usages (most of them for visual separation, e.g., `###########`) @@ -508,37 +499,35 @@ This is why we decided to follow this convention. - Will `nix` itself implement native support like in rust -> `cargo doc`? -- How can a tool keep the connection from where a docstring was defined and where the attribute was exposed (lib/default. Nix exposes mapAttrs which is defined at lib/attrsets. Nix) - - There are more complicated things. - -**Possible answer**: This is a tooling question; implementation details will not be discussed in detail. It is possible to track comments in the AST. As shown before in this PR: [nix/#1652](https://github.com/NixOS/nix/pull/1652) - # Future work [Future]: #future-work +## Migrate the existing comments + +Reformating existing doc-comments but also filtering out those who are not part of a documentation. e.g., Whether they are just empty or contain irrelevant information. + ## Editor support - Implement displaying the related documentation when hovering over an expression. (lspAction/hover) Nix already offers a bunch of LSPs, e.g., [nil](https://github.com/oxalica/nil), [rnix-lsp](https://github.com/nix-community/rnix-lsp) are the most common ones. -## Nixdoc +## nixpkgs Manual tooling -Nixdoc may be changed to differentiate between regular comments and doc-comments. -There might be an intermediate phase of transition, where the old Syntax and features are supported in parallel to allow a phase of transition and refactoring of existing documentation comments. +The current tooling needs to be adopted to this change. With supporting the new format the currently existing scope can be retained to build the nixpkgs manual. -## Documentation generators +## Enhanced Documentation generators We think that a future documentation tool could be out of one of the two following categories. -- (1) Tools that utilize static code analysis and configuration files. +- (1) Tools that utilize static code analysis and configuration files. (This is the current approach) -- (2) Tools that use dynamic evaluation to solve for tracking down name and value relationships and provide more accurate documentation with less configuration overhead. +- (2) Tools that use dynamic evaluation to attach name and value relationships and provides more accurate documentation with less configuration overhead. > We could solve such concerns in [`tvix`](https://tvix.dev/) or in `nix`, which could vend a tool that pre-evaluates expressions and gathers their respective documentation. -For the beginning it seems promising to start with the static approach (1). In the long term dynamic approach (2) seems more promising and accurate but requires a much deeper understanding of compilers and evaluators. -However the border between those two variants is not strict and the community might even build tools that live in between. +For the beginning it seems promising to start with the static approach (1). In the long term a dynamic approach (2) seems more promising and accurate but requires a much deeper understanding of compilers and evaluators. +However the border between those two variants is not strict and we might find future tools that fit our needs just perfectly. ## Type @@ -558,6 +547,7 @@ However the border between those two variants is not strict and the community mi - [Rust](https://doc.rust-lang.org/stable/reference/comments.html#doc-comments) - [Python](https://peps.python.org/pep-0257/) - [JSDoc](https://jsdoc.app/) +- [Go Doc Comments](https://go.dev/doc/comment) ### Related tools From 56f25cade6c96f0961d76dd8f76fbbb234fcfe46 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Fri, 12 May 2023 20:01:58 +0200 Subject: [PATCH 072/110] Add @sternenseemann as sheperd --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index d109d605d..4eb4c8805 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -3,7 +3,7 @@ feature: doc-comment-standard start-date: 2023-03-27 author: hsjobeki co-authors: (find a buddy later to help out with the RFC) -shepherd-team: (@DavHau; tbd.) +shepherd-team: (@DavHau; @sternenseemann; tbd.) shepherd-leader: (name to be appointed by RFC steering committee) related-issues: (will contain links to implementation PRs) --- From be8fa445ff89c40ffbb4f2a5f0f6e7e8611845bb Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Mon, 31 Jul 2023 16:13:09 +0200 Subject: [PATCH 073/110] Add sheperd team members --- rfcs/0145-doc-strings.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 4eb4c8805..5f4ef3ad3 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -2,9 +2,9 @@ feature: doc-comment-standard start-date: 2023-03-27 author: hsjobeki -co-authors: (find a buddy later to help out with the RFC) -shepherd-team: (@DavHau; @sternenseemann; tbd.) -shepherd-leader: (name to be appointed by RFC steering committee) +co-authors: -- +shepherd-team: @DavHau; @sternenseemann; @asymmetric +shepherd-leader: @lassulus related-issues: (will contain links to implementation PRs) --- From bb33f913183f01796e3b8f539913113629b6eb9c Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Mon, 31 Jul 2023 17:32:09 +0200 Subject: [PATCH 074/110] Apply suggestions from @asymmetric Co-authored-by: asymmetric --- rfcs/0145-doc-strings.md | 84 ++++++++++++++++++++++------------------ 1 file changed, 46 insertions(+), 38 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 5f4ef3ad3..6789e47b7 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -11,14 +11,19 @@ related-issues: (will contain links to implementation PRs) # Summary [summary]: #summary -Propose a standard format specification for Doc-comments. +Propose a standard format specification for doc-comments. This RFC includes two concerns that define a doc-comment: - Outer format rules to allow distinction between regular comments and doc-comments -- Inner format rules that describe the required format of the content. +- Inner format rules that describe the required format of a doc-comment. + +However, both concerns relate closely to each other; It makes sense and reduces bureaucracy to address both in a single RFC. +# Definitions + +- **doc-comment**: A comment following a specific structure, intended to document the code's API. +- **internal comment**: A free-form comment whose target audience is someone insterested in the code's implementation. -However, both concerns relate closely to each other; It makes sense and reduces bureaucracy to address that in a single RFC. # Motivation [motivation]: #motivation @@ -31,7 +36,7 @@ The following are the envisioned goals. - E.g. extend with code examples that can be converted to tests (aka doctests). This RFC is a significant change to existing documentation conventions. -It allows distinguishing between regular and doc comments. Having distinction is essential because arbitrary code comments should not end up in generated documentation. +It allows distinguishing between internal and doc comments (see "Definitions" section). Having distinction is essential because internal comments should not end up in generated documentation. > Note: Generating static documentation is a controversial topic in nixpkgs.[^1][^2][^3][^4] We found that it is very hard to generate accurate documentation statically. A correct solution would involve the evaluation of expressions in some way. @@ -40,56 +45,62 @@ It allows distinguishing between regular and doc comments. Having distinction is [^3]: [discourse: users asking how to obtain documentation](https://discourse.nixos.org/t/how-to-generate-documentation-from-arbitrary-nix-code/22292), 2022 [^4]: [discourse: aksing for consensus on documentation](https://discourse.nixos.org/t/any-consensus-on-documentation-generation-tool-for-nixpkgs-manual/15550), 2021 +# Non-goals + +- Discuss in which tool doc-comments are parsed and rendered. This could be an external tool, or Nix, or something else entirely, but that's out of scope for this RFC. +- Providing a migration path for existing comments. This is expected to require some amount of manual work. See "Future work" section. ## Current State -Currently, Nix does not have doc comments. Over time there evolved certain conventions, but they were never formally specified. +Currently, Nix does not have doc comments. Over time some informal conventions have crystallized, but they were never formally specified. Within this RFC, we want to clarify which convention we should use, harmonize and improve the format for doc-comments to achieve the envisioned goals. -We use a tool called [nixdoc](https://github.com/nix-community/nixdoc) to use a 'doc-comment'- like functionality, which allows us to build a subset of static documentation of the `nixpkgs.lib` functions. +A third-party tool called [nixdoc](https://github.com/nix-community/nixdoc) has emerged, which codifies its own rules as to the internal and external formats of a Nix doc-comment. This tool has seen some adoption, notably for the `nixpkgs.lib` functions. The format of a doc-comment was, however, never specified. The nix community does not understand the rules required by `nixdoc`, which we believe is one of the reasons for its lack of adoption. Here is an example of the format understood by *nixdoc*: ```nix -# nixpkgs/lib/attrsets.nix + # nixpkgs/lib/trivial.nix -{ - /* - - Type: - - + /* The constant function + + Ignores the second argument. If called with only one argument, + constructs a function that always returns a static value. + + Type: const :: a -> b -> a Example: - + let f = const 5; in f 10 + => 5 */ - Attr = - # - arg1: - # - arg2: - - # ... implementation -} + const = + # Value to return + x: + # Value to ignore + y: x; ``` ## Current problems -### Lack of having an agreed format +### Multiplicity of formats + +Within nixpkgs alone, several conventions for doc-comments have emerged, see [1], [2] and [3]. -The Nix community interprets the outer format and content of doc-comments differently. Within nixpkgs, several conventions and patterns need to be harmonized. It seems every directory decides upon their own convention; Or doesn't follow any. We immediately observe one clear thing: Most of comments utilize some form of markdown, even if they are not rendered. +Notably, most doc-comments utilize some fraction of the of CommonMark syntax, even if they are not meant to be rendered. + +[1]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/trivial-builders/default.nix +[2]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix +[3]: https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-disk-image.nix In general, the format for writing documentation strings is **not formally specified**. -The *nixdoc*-tool requires a somewhat consistent format, but the basic format was never specified and cannot be enforced in parts of nixpkgs where nixdoc is currently not applied. +Among the formats encountered in the wild, the one used in `nixpkgs/lib` is the only one intended to be rendered as part of an API documentation, via the nixdoc third-party tool, whose syntax has not been standardized. Extending the scope of *nixdoc* is not the primary goal. Instead, we should find formal rules for writing *doc-comments*. Tools like *nixdoc* can then implement against this RFC instead of the format relying on nixdoc implementation details. -### Impossible to differentiate from regular comments - -The format does not allow any distinction between doc comments and regular comments. +### Impossible to differentiate from internal comments -Having a distinction would allow us to build reliable and correct documentation +The lack of a formal definition of a doc-comment also means there is no reliable way to distinguish them from internal comments, which would result in automatically-produced API documentation which includes the wrong type of comments. ### References to the problems above @@ -436,18 +447,15 @@ Some more ## Changes the existing comments inside the code base -This could be automated. (e.g., via codemod) - -Also, this affects only the `lib` directory and a few other places currently used to build the documentation. +This could be automated. -## Changes in the `nixdoc` tooling are required +Also, the migration could be performed piecemal, starting perhaps with `nixpkgs.lib`, as it is already parsed by a tool (nixdoc), which could be modified for the new standard. -It remains an open question that needs to be tried; +## Requires changes in existing tooling to produce documentation -Can we still build the current Nixos manuals with the new standard? +nixdoc, the tool used to produce Nixpkgs library function documentation, would have to be modified to fit the new format. -- We can wait to change everything. -- Migration can happen in small steps. +This would be a small change, and anyway this RFC is agnostic to the tool used -- an entirely new tool could be developed, or the functionality be included as part of Nix. # Alternatives [alternatives]: #alternatives @@ -504,7 +512,7 @@ This is why we decided to follow this convention. ## Migrate the existing comments -Reformating existing doc-comments but also filtering out those who are not part of a documentation. e.g., Whether they are just empty or contain irrelevant information. +Reformatting existing doc-comments in Nixpkgs, but also filtering out false-positives, i.e. those that should not be part of the API documentation. ## Editor support From 63321c4801348d6c502f77074d918f1574ce1c34 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Mon, 31 Jul 2023 17:41:17 +0000 Subject: [PATCH 075/110] Rework: improves content for problem targeting and readablity --- rfcs/0145-doc-strings.md | 106 ++++++++++++++++++--------------------- 1 file changed, 48 insertions(+), 58 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 6789e47b7..05a0a4cc5 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -21,42 +21,39 @@ This RFC includes two concerns that define a doc-comment: However, both concerns relate closely to each other; It makes sense and reduces bureaucracy to address both in a single RFC. # Definitions -- **doc-comment**: A comment following a specific structure, intended to document the code's API. -- **internal comment**: A free-form comment whose target audience is someone insterested in the code's implementation. +For this RFC, we adopt the following definitions: +- **doc-comment**: A structured comment documenting the code's API. +- **internal comment**: A free-form comment for those interested in the code's implementation. + +The doc-comment properties are grouped into these subcategories: + +- **Outer format**: Specifies rules linking code (API) and doc-comments. (e.g. placement, syntax rules) + +- **Inner format**: Specifies rules affecting the comment's actual content. (e.g. usage of commonMark) # Motivation [motivation]: #motivation The following are the envisioned goals. -- Be able to generate documentation from code for any nix expression. -- Make doc comments easy to write and read -- Standardize a format for doc comments that further RFCs can extend - - E.g. extend with code examples that can be converted to tests (aka doctests). - -This RFC is a significant change to existing documentation conventions. -It allows distinguishing between internal and doc comments (see "Definitions" section). Having distinction is essential because internal comments should not end up in generated documentation. +- Create distinct outer and inner formats for Nix doc-comments to enable accurate automated parsing and extraction. +- Ensure clear differentiation from internal comments, making them accessible to tooling solutions such as documentation rendering and building. -> Note: Generating static documentation is a controversial topic in nixpkgs.[^1][^2][^3][^4] We found that it is very hard to generate accurate documentation statically. A correct solution would involve the evaluation of expressions in some way. +- Switch existing Nixpkgs code to this format. -[^1]: [Unmerged PR: nix repl doc-comments](https://github.com/NixOS/nix/pull/1652), 2017 -[^2]: [Unmerged PR: doc-strings for functors](https://github.com/NixOS/nix/pull/5527), 2021 -[^3]: [discourse: users asking how to obtain documentation](https://discourse.nixos.org/t/how-to-generate-documentation-from-arbitrary-nix-code/22292), 2022 -[^4]: [discourse: aksing for consensus on documentation](https://discourse.nixos.org/t/any-consensus-on-documentation-generation-tool-for-nixpkgs-manual/15550), 2021 +- In addition, the developer experience and adherence to established conventions should be taken into account. Equally important is ensuring that doc comments remain effortless to compose and comprehend, though it is essential to acknowledge that this aspect may vary subjectively based on personal preferences. # Non-goals - Discuss in which tool doc-comments are parsed and rendered. This could be an external tool, or Nix, or something else entirely, but that's out of scope for this RFC. - Providing a migration path for existing comments. This is expected to require some amount of manual work. See "Future work" section. -## Current State -Currently, Nix does not have doc comments. Over time some informal conventions have crystallized, but they were never formally specified. -Within this RFC, we want to clarify which convention we should use, harmonize and improve the format for doc-comments to achieve the envisioned goals. +- Extending the scope of *nixdoc* is not a goal. Instead, this RFC finds formal rules for writing *doc-comments*. Tools like *nixdoc* can then implement against this RFC. -A third-party tool called [nixdoc](https://github.com/nix-community/nixdoc) has emerged, which codifies its own rules as to the internal and external formats of a Nix doc-comment. This tool has seen some adoption, notably for the `nixpkgs.lib` functions. +## Current State -The format of a doc-comment was, however, never specified. The nix community does not understand the rules required by `nixdoc`, which we believe is one of the reasons for its lack of adoption. +A third-party tool called [nixdoc](https://github.com/nix-community/nixdoc) has emerged, which codifies its own rules as to the internal and external formats of a Nix doc-comment. This tool has seen some adoption, notably for the `nixpkgs.lib` functions. Here is an example of the format understood by *nixdoc*: @@ -96,15 +93,13 @@ In general, the format for writing documentation strings is **not formally speci Among the formats encountered in the wild, the one used in `nixpkgs/lib` is the only one intended to be rendered as part of an API documentation, via the nixdoc third-party tool, whose syntax has not been standardized. -Extending the scope of *nixdoc* is not the primary goal. Instead, we should find formal rules for writing *doc-comments*. Tools like *nixdoc* can then implement against this RFC instead of the format relying on nixdoc implementation details. - ### Impossible to differentiate from internal comments The lack of a formal definition of a doc-comment also means there is no reliable way to distinguish them from internal comments, which would result in automatically-produced API documentation which includes the wrong type of comments. ### References to the problems above -> Note: That specific collection of links should show the amount of inconsistency within the Nix ecosystem, which is a target of this RFC. +> This curated link collection highlights the Nix ecosystem's inconsistencies, a primary focus of this RFC. #### nixpkgs - doc-comment examples @@ -113,46 +108,34 @@ The lack of a formal definition of a doc-comment also means there is no reliable - [stdenv/mkDerivation](https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix) - [nixos/lib/make-disk-image](https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-disk-image.nix) -#### Current tools +# Design +[design]: #detailed-design -We collected some Tools that work directly with the nix AST and comments. -Those would benefit from a standardized doc-comment. +In the following we give a comprehensive overview to our decision that we've made. Detailed arguments for and against every decision can be found in the [Decisions](#Decisions) section -- [noogle](https://noogle.dev) - Nix API search engine. It allows searching functions and other expressions. -- [nix-doc](https://github.com/lf-/nix-doc) - A Nix developer tool leveraging the rnix Nix parser for intelligent documentation search and tags generation -- [manix](https://github.com/mlvzk/manix) - A fast CLI documentation searcher for Nix. +## CommonMark -# Detailed design -[design]: #detailed-design +**CommonMark is the content of all doc-comments.** -Each subsection here contains a decision along with arguments and counter-arguments for (+) and against (-) that decision. +Adopting CommonMark as the content for all doc-comments brings the benefits of widely accepted and understood documentation format in tech projects, while maintaining profitability and consistency within the Nix ecosystem by aligning with existing [NixOS/RFC-72](https://github.com/NixOS/rfcs/blob/master/rfcs/0072-commonmark-docs.md). -> Note: This RFC does not promote any tool; It focuses only on the format, not implementation. At the same time, this RFC must be technically feasible. +## `/**` starts a doc-comment -## Discontinue nixdoc's custom format +The decision to use /** to start a doc-comment ensures a unique distinction from regular comments while still allowing seamless writing without IDE or editor support. This choice not only provides the best developer experience but also minimizes the need for additional tooling overhead. -**Observing**: Nixdoc currently provides some format that can generate documentation from a small fraction of files in nixpkgs, Namely the files for `.lib` functions. The directory provides a consistent community convention that allows the building of static documentation. Applying nixdoc to files outside that directory is only partially possible due to the lack of a standard doc-comment format. +## Placement -**Considering**: Should we propose the `nixdoc` format as a community standard? +> TODO: Finding a universal approach for describing the appropriate placement that effectively links doc-comments with corresponding code elements. -**Decision**: Discontinue nixdoc's custom format in favor of using something more generic. -
-Arguments -- (+) Stays primarily compatible with the currently used comments in lib. - - (-) Outside the ./lib directory, nixodc is not a convention. -- (-) Is not widely understood. -- (-) The format is specific, should be more generic, and follow established conventions. (e.g., Markdown, Doxygen, JSdoc, Haddock) -- (-) The format is too complex and needs a whole list of complex rules to describe the exact format and content rules; this can also be observed in many false-formatted comments that do not render correctly. - - (-) The sections are implicitly continued. - - (-) The predefined section headers/keywords are very close to natural language and may accidentally be used. - - (-) Syntax to start new sections does not follow conventions (e.g., `Example:`) In contrast JSdoc defines `@Example`; Markdown `# Example`; Doxygen `@example`; Haskell-Haddock `>>>` - - (-) The content is implicitly rendered differently depending on predefined rules. Those rules need to be understood/feel natural. -- (+) Some usages outside of /lib directory indicate the process of adoption. -- (-) Some multiline comments become doc-comments, but it is unclear which comments have such a property. - -
+ +# Decisions + + +Each subsection here contains a decision along with arguments and counter-arguments for (+) and against (-) that decision. + +> Note: This RFC does not promote any tool; It focuses only on the format, not implementation. At the same time, this RFC must be technically feasible. ## `/**` to start a doc-comment @@ -229,15 +212,15 @@ Each subsection here contains a decision along with arguments and counter-argume
-## Place doc-comments before the referenced expression +## TODO: Placement of doc-comments -**Observing**: Doc-comments currently are placed above the expression they document. More precisely, only named attribute bindings `foo = ` can be documented. There is also the need to support documentation for anonymous functions. More generally, there is the need to document anonymous expressions where they are defined. +> This decision is still under construction by the rfc shepherd team and needs a final proof of concept implementaion. -**Considering**: General reference logic between doc-comments and expressions. +**Observing**: Doc-comments currently are placed above the expression they document. More precisely, only named attribute bindings `foo = ` can be documented. There is also the need to support documentation for anonymous functions. More generally, it would be desirable to document anonymous expressions. -**Decision**: Doc-comment refer to the expression that is **immediately** next to it. Only whitespaces are allowed between doc-comment and following expression. +**Considering**: General linking logic between doc-comments and expressions / values. Taking into account both static and dynamic implementation requirements. -> Note: We are working on a generic example implementation that covers about 90% of all documentation use cases. +**Decision**: Doc-comment refer to the expression that is **immediately** next to it. Only whitespaces are allowed between doc-comment and following expression.
Arguments @@ -565,6 +548,13 @@ However the border between those two variants is not strict and we might find fu ## Related discussions -- [@documentation-team (meet @ 13.Apr 2023)](https://discourse.nixos.org/t/2023-04-13-documentation-team-meeting-notes-41/27264) + - https://discourse.nixos.org/t/2023-04-13-documentation-team-meeting-notes-41/27264) + +- https://github.com/NixOS/nix/issues/3904 +- https://github.com/NixOS/nix/issues/228 + + +- https://github.com/NixOS/nix/pull/5527 +- https://github.com/NixOS/nix/pull/1652 -- There is an actual but rather old PR (@roberth) that uses just comments to show documentation in the nix repl for functions. -> https://github.com/NixOS/nix/pull/1652 +[#1652](https://github.com/NixOS/nix/pull/1652) gets closest what this RFC imagines, implementation is almost done. It certainly has some limitations, but got stuck on politics and diverging opinions about markdown and syntax rules. From f1468071508668405665a06a58847bfeb3b89a3c Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 28 Sep 2023 20:12:29 +0200 Subject: [PATCH 076/110] Clean up RFC according to meeting notes --- rfcs/0145-doc-strings.md | 404 +++++++++++++++++---------------------- 1 file changed, 173 insertions(+), 231 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 05a0a4cc5..c399f15ac 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -19,6 +19,7 @@ This RFC includes two concerns that define a doc-comment: - Inner format rules that describe the required format of a doc-comment. However, both concerns relate closely to each other; It makes sense and reduces bureaucracy to address both in a single RFC. + # Definitions For this RFC, we adopt the following definitions: @@ -40,16 +41,13 @@ The following are the envisioned goals. - Create distinct outer and inner formats for Nix doc-comments to enable accurate automated parsing and extraction. - Ensure clear differentiation from internal comments, making them accessible to tooling solutions such as documentation rendering and building. -- Switch existing Nixpkgs code to this format. +- Migrate `nixpkgs'` comments to this format. - In addition, the developer experience and adherence to established conventions should be taken into account. Equally important is ensuring that doc comments remain effortless to compose and comprehend, though it is essential to acknowledge that this aspect may vary subjectively based on personal preferences. # Non-goals - Discuss in which tool doc-comments are parsed and rendered. This could be an external tool, or Nix, or something else entirely, but that's out of scope for this RFC. -- Providing a migration path for existing comments. This is expected to require some amount of manual work. See "Future work" section. - -- Extending the scope of *nixdoc* is not a goal. Instead, this RFC finds formal rules for writing *doc-comments*. Tools like *nixdoc* can then implement against this RFC. ## Current State @@ -101,7 +99,7 @@ The lack of a formal definition of a doc-comment also means there is no reliable > This curated link collection highlights the Nix ecosystem's inconsistencies, a primary focus of this RFC. -#### nixpkgs - doc-comment examples +#### nixpkgs - comment examples - [lib/attrsets](https://github.com/NixOS/nixpkgs/blob/master/lib/attrsets.nix) - [trivial-builders](https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/trivial-builders.nix) @@ -119,19 +117,157 @@ In the following we give a comprehensive overview to our decision that we've mad Adopting CommonMark as the content for all doc-comments brings the benefits of widely accepted and understood documentation format in tech projects, while maintaining profitability and consistency within the Nix ecosystem by aligning with existing [NixOS/RFC-72](https://github.com/NixOS/rfcs/blob/master/rfcs/0072-commonmark-docs.md). -## `/**` starts a doc-comment +The content indentation inherits the indentation logic from nixs' multiline strings `'' ''` allowing any arbitrary indentation the user might prefer. + +## `/** */` is the doc-comment format The decision to use /** to start a doc-comment ensures a unique distinction from regular comments while still allowing seamless writing without IDE or editor support. This choice not only provides the best developer experience but also minimizes the need for additional tooling overhead. ## Placement -> TODO: Finding a universal approach for describing the appropriate placement that effectively links doc-comments with corresponding code elements. +- Documentable nodes are `lambda value` and `attribute name` only. (This might be extended in the future) +- Doc-comments are placed before the documentable node. Only `WHITESPACES` are allowed in between. + - `WHITESPACES` are: `[\n \r ' ' \t]`, singleline- and multiline-comments (that are not doc-comments). +Accurate and usable Lambda documentation reqires some more details to be specified: +- **If a lambda is assigned to an `attribute name`, then the `attribute name`-documentation is also the `lambda value` documentation**. + - In case both places have doc-comments, the one closer to the lambda has higher precedence. + +- In case of curried functions all partial functions can share the same placement with the outermost lambda. -# Decisions +> Note: Research of the RFC Sheperds Team showed That this allows for intuitive placements like are already done in nixpkgs. + +### Lambda Examples + +```nix +/**Doc for anonymous lambda function*/ +↓ +x: x; + +# doc -> Lambda Value '(x: x)' +``` + +```nix +/**Doc for assigned lambda function*/ +↓ +assigned = x: x; +``` + +```nix +/**Doc B*/ +assigned = /**Doc A*/x: x; + +# Lambda Documentation is 'Doc A' because it is directly next to the documentable lambda. +``` + +### Attribute Examples + +```nix +{ + /** Documentation for 'foo' */ + ↓ + foo = 1; +} +# doc -> Attribute Name foo +``` + +```nix +{ + /** Documentation for 'bar' */ + ↓ + ${let name = "bar"; in name} = 2; +} +# Interpolated attribute names +``` + +```nix +{ + /** Documentation for 'bar' */ + ↓ + ${let name = "bar"; in name} = 2; +} +# Interpolated attribute names +``` + +```nix +setWithMappedNamed = + lib.mapAttrs' (name: value: { + /**Undocumentable attribute name*/ + ↓ + name = "${n + "foo"}"; + inherit value; + }) + { + a = 1; + b = 2; + }; +# Dynamically constructed attribute names cannot be documented. (constructed in a function call like above) +``` + +### Examples when the lambda is not directly assigned + +```nix +{ + /**This documents (1), NOT (2) */ + ↓1 ↓2 + foo = map (x: x); + ^~~~~~~~~~~ Evaluates to a specialisation of map, which is a partially applied lambda and defined elsewhere. + +# IMPORTANT: Documentation of the specialized map function can still be done as shown. But is related to attrName 'foo' (1) +} +``` + +```nix +{ + /**This documents (1), NOT (2) */ + ↓1 ↓2 + foo = mapAttrs (name: value: (x: x) ) {a=1; b=2;}; + # => + # foo = { + # a = x: x; + # b = x: x; + # } +# Doesn't document the inner anonymous lambda function. +} +``` + +### Partially applied lambda + +All partial functions can share the same placement with the outermost lambda. + +```nix +/**Generic doc for all partial lambda functions too*/ +↓1 ↓2 ↓3 +x: ({y, ...}: z: x + y) * z; + +# 1 -> Lambda Value `x: ({y, ...}: z: x + y) * z` countApplied: 0 +# 2 -> Lambda Value `({y, ...}: z: x + y) * z` countApplied: 1 +# 3 -> Lambda Value `z: x + y) * z` countApplied: 2 +``` +```nix +{ + /** incrMap. Adds one to every item. */ + ↓ + incrMap = map (x: x + 1); + ^~~~~~~~~~~ Evaluates to a specialisation of map, which is a partially applied lambda and defined elsewhere. + +# The lambda documementation is taken from the place where the `map` lambda is defined +# Retrieve the documentation from the attribute 'incrMap'. +} +``` + +Still all other 'Fundamentals' apply. + +```nix + Always doc for anonymous lambda function 'y: x + y' + ↓ +x: /** Partial Doc for y: x+y */ y: x + y; +``` + +# Decisions Each subsection here contains a decision along with arguments and counter-arguments for (+) and against (-) that decision. @@ -141,7 +277,7 @@ Each subsection here contains a decision along with arguments and counter-argume **Observing**: Doc-comments' outer format should be a distinctive subset of regular comments. Nevertheless, it should allow native writing without an IDE or editor support. -**Considering**: `/** {content} */` where `{content}` is the inner format which is discussed later. +**Considering**: `/** {content} */` where `{content}` is the inner format. **Decision**: use `/** {content} */` as the outer format. @@ -212,219 +348,32 @@ Each subsection here contains a decision along with arguments and counter-argume
-## TODO: Placement of doc-comments - -> This decision is still under construction by the rfc shepherd team and needs a final proof of concept implementaion. - -**Observing**: Doc-comments currently are placed above the expression they document. More precisely, only named attribute bindings `foo = ` can be documented. There is also the need to support documentation for anonymous functions. More generally, it would be desirable to document anonymous expressions. - -**Considering**: General linking logic between doc-comments and expressions / values. Taking into account both static and dynamic implementation requirements. - -**Decision**: Doc-comment refer to the expression that is **immediately** next to it. Only whitespaces are allowed between doc-comment and following expression. - -
-Arguments - -- (+) Doc-comments should have only one variant to reduce complexity. Referencing the next node seems straightforward. -- (-) A variant that references the previous node in the AST should be avoided to reduce complexity. -- (+) Relation between documentation and the referenced implementation is straightforward and back-trackable. -- (-) Concrete Implementation might be complex. -- (-) Unclear if complete documentation might also need backward references. - - (-), e.g., rust uses backward references only at the top of file comments. - - (+) Nix files can only have ONE expression; the next AST Node, in case of top-of-file comments, is thus always only that one expression. (Unlike in Rust) -- (-) Tools need to be smart enough to understand asignments `=` and other forms of creating names for anonymous expressions. (e.g., `callPackage` and `import` ) - - (+) Tools can still come up with other solutions that do not involve calculating everything dynamically from nix code but could also involve a static configuration. - - (+) The whole `tool point` is an implementation detail as long as it is not impossible. The current tool, `nixdoc`, already proves that it is possible to have static documentation to a certain degree. - -
- ## Single-line doc-comments (do not exist) **Observing**: Nix offers two variants of comments; single- (`#`) and multi-line comments (`/* */`). There may be use cases where it is desirable to have a form of single-line comments subtyped for doc-comment purposes. -**Considering**: Single-line comment for documentation. (Starting a doc-comment with `#`) +**Considering**: Single-line comment for documentation. (Starting a doc-comment with `##`) -**Decision**: Single-line comments (starting with `#`) **cannot be used** in any form for documentation puposes. +**Decision**: Single-line comments (starting with `##`) **cannot be used** in any form for documentation puposes.
Arguments +- (+) It Would be consistent with providing variants for both nix comments. - (-) Doc-comments should have only one variant to reduce complexity. -- (-) documentation will likely take up more than one line. +- (-) documentation will likely everytime take up more than one line. - (-) If documentation grows bigger than one line, refactoring into a multiline-doc-comment must occur. -- (+) It Would be consistent with providing variants for both nix comments. - (+) Offer the choice. -- (o) Single lines can also be concatenated to form multi-line documentation. +- (o) Single lines could also be concatenated to form multi-line documentation. + - (+) Convinience + - (-) Technical more complex - (+) Takes up less vertical space -- (-) Visually confusing when every line starts with a reserved character. - - (-) Potential visual conflicts with the content +- (-) Visually confusing when every line starts with a `#` character. + - (-) Potential visual conflicts with the content that is markdown (e.g. with headings). - (+) Indentation of the content is clear.
-## Doc-comment examples - -This section contains many examples for some different use cases we; Visualize them and emphasize the previously discussed characteristics. - -`somefile.nix` - -````nix -{ - /** - Documentation for the fundamental 'id' function - - # Examples - - ``` - id 1 - => - 1 - ``` - - # Type - - ``` - id :: a -> a - ``` - - */ - id = x: x; -} -```` - -### Attribute bindings - -`somefile.nix` - -````nix - { - /** - - mapAttrs is a well-known function - - # Examples - - ``` - # some code examples - ``` - - */ - mapAttrs = f: s: #... - } -```` - -### Indentation follows know nix's `''` behavior - -Indentation follows the know `''`-nix multiline strings behavior. -Making usage more intuitive and and one point less to think about. - -````nix - { - /** - Line 1 - Line 2 - */ - id = f: s: #... - } -```` --> -```markdown -Line 1 - Line 2 -``` - -Or more advanced - -````nix - { - /** - # Title - - Some foo bar - - ## Subtitle - - Some more - - indented 1 - - indented 2 - - */ - id = f: s: #... - } -```` -=> -```markdown -# Title - -Some foo bar - -## Subtitle - -Some more - - indented 1 - - indented 2 - -``` - - -### NixOS Module documentation - -> Note: NixOS Modules also can produce documentation from their interface declarations. However, this does not include a generic description and usage examples. - -`myModule.nix` -````nix -/** - This is a custom module. - - It configures a systemd service. - - # Examples - - Different use case scenarios - how to use this module -*/ -{config, ...}: -{ - config = f: s: f s; -} -```` - -### Anonymous function documentation - -`function.nix` -````nix -/** - This is an anonymous function implementation. - - It does not have a name yet. - Nevertheless, documentation can be right next to the implementation. - - The name gets assigned later. -*/ -{a, b}: -{ - sum = a + b; -} -```` - -### Anonymous expression documentation - -`exp.nix` -````nix -/** - This is an anonymous string. - - It is the documentation for the "anonymous expression" use case. - - Although this example is relatively superficial, there might be use cases. -*/ -"-p=libxml2/include/libxml2" -```` - # Drawbacks [drawbacks]: #drawbacks @@ -432,13 +381,15 @@ Some more This could be automated. -Also, the migration could be performed piecemal, starting perhaps with `nixpkgs.lib`, as it is already parsed by a tool (nixdoc), which could be modified for the new standard. +Also, the migration could be performed piecemal, starting perhaps with `nixpkgs.lib`. -## Requires changes in existing tooling to produce documentation +See future work. -nixdoc, the tool used to produce Nixpkgs library function documentation, would have to be modified to fit the new format. +## Tooling to produce the nixpkgs manual -This would be a small change, and anyway this RFC is agnostic to the tool used -- an entirely new tool could be developed, or the functionality be included as part of Nix. +Change or write a new tool used to produce Nixpkgs library function documentation. Because this is a breaking change. + +See future work. # Alternatives [alternatives]: #alternatives @@ -465,10 +416,10 @@ This would be a small change, and anyway this RFC is agnostic to the tool used - nixpkgs comments: -- `##` ~4k usages (most of them for visual separation, e.g., `###########`) +- `##` ~4k usages - `#` ~20k usages - `/*` ~6k usages -- `/**` 160 usages (most empty ?) +- `/**` 160 usages Choosing `/**` or subsets would cause minor conflicts within current nixpkgs. While this is NOT the main reason for the final decision, it MUST be considered. @@ -476,14 +427,13 @@ Choosing `/**` or subsets would cause minor conflicts within current nixpkgs. Wh While this allows the most freedom, it is usually considered the best option, not creating any restrictions. -However, [RFC72](https://github.com/NixOS/rfcs/blob/master/rfcs/0072-commonmark-docs.md) defines commonMark as the official documentation format. -This is why we decided to follow this convention. +- [RFC72](https://github.com/NixOS/rfcs/blob/master/rfcs/0072-commonmark-docs.md) defines commonMark as the official documentation format. +This is why we decided to follow this convention. Writing plain text is still possible. ## Consequences of not implementing this - By not implementing this feature, Nix gains no ability for tool-generated documentation. -- Documentation will be defined by nixdoc, not by the nix community. -- Many existing comments written for documentation will remain imperceptible. +- Many existing comments written for documentation will remain imperceptible for automated tooling. # Unresolved questions [unresolved]: #unresolved-questions @@ -497,12 +447,6 @@ This is why we decided to follow this convention. Reformatting existing doc-comments in Nixpkgs, but also filtering out false-positives, i.e. those that should not be part of the API documentation. -## Editor support - -- Implement displaying the related documentation when hovering over an expression. (lspAction/hover) - -Nix already offers a bunch of LSPs, e.g., [nil](https://github.com/oxalica/nil), [rnix-lsp](https://github.com/nix-community/rnix-lsp) are the most common ones. - ## nixpkgs Manual tooling The current tooling needs to be adopted to this change. With supporting the new format the currently existing scope can be retained to build the nixpkgs manual. @@ -511,25 +455,23 @@ The current tooling needs to be adopted to this change. With supporting the new We think that a future documentation tool could be out of one of the two following categories. -- (1) Tools that utilize static code analysis and configuration files. (This is the current approach) +- (1) Tools that utilize static code analysis and configuration files. - (2) Tools that use dynamic evaluation to attach name and value relationships and provides more accurate documentation with less configuration overhead. - > We could solve such concerns in [`tvix`](https://tvix.dev/) or in `nix`, which could vend a tool that pre-evaluates expressions and gathers their respective documentation. - For the beginning it seems promising to start with the static approach (1). In the long term a dynamic approach (2) seems more promising and accurate but requires a much deeper understanding of compilers and evaluators. -However the border between those two variants is not strict and we might find future tools that fit our needs just perfectly. -## Type +However the border between those two variants is not strict and we might find future tools that fit our needs just perfectly. -- An RFC under construction specifies the used Syntax within the `# Type` Heading. -- The `type` feature should belong in the nix syntax. Try them within the comments first; This is still possible. +## Native support in Nix -- see a [preview](https://typednix.dev) of an eventual future doc-type-syntax. +- `NixOS/nix` should implement native support for doc-comments so that our users do not have to rely on nixpkgs or external tools. Those tools can still exist and provide more custom functionality, but documenting nix expressions should be natively possible. -## Native support in Nix +We propose to add: -- `NixOS/nix` could implement native support for doc-comments so that our users do not have to rely on nixpkgs or external tools. Those tools can still exist and provide more custom functionality, but documenting nix expressions should be natively possible. +- Adding two builtins: `getAttrDoc` and `getLambdaDoc` +- both introduced with `unsafe` prefix +- until properly stabilized according to the official process for feature stabilization. ## References From 4c8fc55f66dbdf0b29b0bce590e5bcc3f91c0fb6 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Fri, 29 Sep 2023 10:06:18 +0200 Subject: [PATCH 077/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 138 +++++++++++---------------------------- 1 file changed, 39 insertions(+), 99 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index c399f15ac..ccb6cb4b0 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -125,146 +125,86 @@ The decision to use /** to start a doc-comment ensures a unique distinction from ## Placement -- Documentable nodes are `lambda value` and `attribute name` only. (This might be extended in the future) +**The placment describes the relationship between doc-comments and the expression they are documenting.** - Doc-comments are placed before the documentable node. Only `WHITESPACES` are allowed in between. - - `WHITESPACES` are: `[\n \r ' ' \t]`, singleline- and multiline-comments (that are not doc-comments). + - `WHITESPACES` are: `[\n \r ' ' \t]`. -Accurate and usable Lambda documentation reqires some more details to be specified: +- The documentation present before the `attribute path` describes the body of the attribute. + + [Examples](#Examples) + - In case placement is ambiguous, the one closer to the body has higher precedence. -- **If a lambda is assigned to an `attribute name`, then the `attribute name`-documentation is also the `lambda value` documentation**. - - In case both places have doc-comments, the one closer to the lambda has higher precedence. + [Ambiguous placement](#Ambiguous-placement) -- In case of curried functions all partial functions can share the same placement with the outermost lambda. +- All partial functions of a curried lambda can share the same placement with the outermost lambda. + + [partial lambda functions](#partial-lambda-functions) + -> Note: Research of the RFC Sheperds Team showed That this allows for intuitive placements like are already done in nixpkgs. +> Note: Research of the RFC Sheperds Team showed that this allows for intuitive placements like are already done in nixpkgs. -### Lambda Examples +### Examples ```nix /**Doc for anonymous lambda function*/ ↓ x: x; - -# doc -> Lambda Value '(x: x)' ``` ```nix -/**Doc for assigned lambda function*/ -↓ +/**Doc for lambda function bound to a variable*/ + ↓ assigned = x: x; ``` -```nix -/**Doc B*/ -assigned = /**Doc A*/x: x; - -# Lambda Documentation is 'Doc A' because it is directly next to the documentable lambda. -``` - -### Attribute Examples - ```nix { - /** Documentation for 'foo' */ - ↓ - foo = 1; + /**This documents the specialisation `map (x: x)` */ + ↓ + foo = map (x: x); } -# doc -> Attribute Name foo ``` ```nix -{ - /** Documentation for 'bar' */ - ↓ - ${let name = "bar"; in name} = 2; -} -# Interpolated attribute names +builtins.listToAttrs [ + { name = "foo"; value = /**Documentation for '1'*/1; } +] +# => +# { foo = 1; } ``` -```nix -{ - /** Documentation for 'bar' */ - ↓ - ${let name = "bar"; in name} = 2; -} -# Interpolated attribute names -``` +#### Ambiguous placement ```nix -setWithMappedNamed = - lib.mapAttrs' (name: value: { - /**Undocumentable attribute name*/ - ↓ - name = "${n + "foo"}"; - inherit value; - }) - { - a = 1; - b = 2; - }; -# Dynamically constructed attribute names cannot be documented. (constructed in a function call like above) -``` - -### Examples when the lambda is not directly assigned +/**Doc B*/ +int = /**Doc A*/1; -```nix -{ - /**This documents (1), NOT (2) */ - ↓1 ↓2 - foo = map (x: x); - ^~~~~~~~~~~ Evaluates to a specialisation of map, which is a partially applied lambda and defined elsewhere. - -# IMPORTANT: Documentation of the specialized map function can still be done as shown. But is related to attrName 'foo' (1) -} +# Documentation is 'Doc A' because it is directly next to the documentable body. ``` +#### Dynamic attribute + ```nix { - /**This documents (1), NOT (2) */ - ↓1 ↓2 - foo = mapAttrs (name: value: (x: x) ) {a=1; b=2;}; - # => - # foo = { - # a = x: x; - # b = x: x; - # } -# Doesn't document the inner anonymous lambda function. + /** Documentation for '2' */ + ↓ + ${let name = "bar"; in name} = 2; } +# Dynamic attribute ``` -### Partially applied lambda - -All partial functions can share the same placement with the outermost lambda. +#### Partial lambda functions ```nix -/**Generic doc for all partial lambda functions too*/ +/**Generic doc for all partial lambda functions*/ ↓1 ↓2 ↓3 x: ({y, ...}: z: x + y) * z; -# 1 -> Lambda Value `x: ({y, ...}: z: x + y) * z` countApplied: 0 -# 2 -> Lambda Value `({y, ...}: z: x + y) * z` countApplied: 1 -# 3 -> Lambda Value `z: x + y) * z` countApplied: 2 -``` - -```nix -{ - /** incrMap. Adds one to every item. */ - ↓ - incrMap = map (x: x + 1); - ^~~~~~~~~~~ Evaluates to a specialisation of map, which is a partially applied lambda and defined elsewhere. - -# The lambda documementation is taken from the place where the `map` lambda is defined -# Retrieve the documentation from the attribute 'incrMap'. -} -``` - -Still all other 'Fundamentals' apply. - -```nix - Always doc for anonymous lambda function 'y: x + y' - ↓ -x: /** Partial Doc for y: x+y */ y: x + y; +# All partial functions can share the same placement with the outermost lambda. +# 1 -> Lambda Value `x: ({y, ...}: z: x + y) * z` +# 2 -> Lambda Value `({y, ...}: z: x + y) * z` +# 3 -> Lambda Value `z: x + y) * z` ``` # Decisions From 273ace0541db9a70d0152cfaee7244e54d84be9b Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Fri, 29 Sep 2023 10:13:50 +0200 Subject: [PATCH 078/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index ccb6cb4b0..d0aca55df 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -129,29 +129,41 @@ The decision to use /** to start a doc-comment ensures a unique distinction from - Doc-comments are placed before the documentable node. Only `WHITESPACES` are allowed in between. - `WHITESPACES` are: `[\n \r ' ' \t]`. + - [Basic examples](#basic-examples) - The documentation present before the `attribute path` describes the body of the attribute. - - [Examples](#Examples) + - [Ambiguous placement](#ambiguous-placement) - In case placement is ambiguous, the one closer to the body has higher precedence. - [Ambiguous placement](#Ambiguous-placement) - All partial functions of a curried lambda can share the same placement with the outermost lambda. - - [partial lambda functions](#partial-lambda-functions) + - [partial lambda functions](#partial-lambda-functions) > Note: Research of the RFC Sheperds Team showed that this allows for intuitive placements like are already done in nixpkgs. ### Examples +#### Basic examples + +Only whitespaces between the `documentable node` and the `doc-comment` + ```nix /**Doc for anonymous lambda function*/ ↓ x: x; ``` +```nix +listToAttrs [ + { name = "foo"; value = /**Documentation for '1'*/1; } +] +``` + +#### Attributes + +It is allowed to write the documentation before the attribute instead of placing it right before the body. + ```nix /**Doc for lambda function bound to a variable*/ ↓ @@ -166,14 +178,6 @@ assigned = x: x; } ``` -```nix -builtins.listToAttrs [ - { name = "foo"; value = /**Documentation for '1'*/1; } -] -# => -# { foo = 1; } -``` - #### Ambiguous placement ```nix From ba8d6f57b51ae9833527146b33feb886057c2cab Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Fri, 29 Sep 2023 10:15:46 +0200 Subject: [PATCH 079/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index d0aca55df..48bb68cd5 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -129,15 +129,16 @@ The decision to use /** to start a doc-comment ensures a unique distinction from - Doc-comments are placed before the documentable node. Only `WHITESPACES` are allowed in between. - `WHITESPACES` are: `[\n \r ' ' \t]`. - - [Basic examples](#basic-examples) + - Example: [Basic examples](#basic-examples) - The documentation present before the `attribute path` describes the body of the attribute. - - [Ambiguous placement](#ambiguous-placement) +- Example: [Attributes](#Attributes) - In case placement is ambiguous, the one closer to the body has higher precedence. + - Example: [Ambiguous placement](#ambiguous-placement) - All partial functions of a curried lambda can share the same placement with the outermost lambda. - - [partial lambda functions](#partial-lambda-functions) + - Example: [partial lambda functions](#partial-lambda-functions) > Note: Research of the RFC Sheperds Team showed that this allows for intuitive placements like are already done in nixpkgs. From 55993447f719ea95ba0d4953f58626a3ab5b7b2d Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Fri, 29 Sep 2023 10:52:43 +0200 Subject: [PATCH 080/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 79 ++++++++++++++-------------------------- 1 file changed, 27 insertions(+), 52 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 48bb68cd5..3bc3fb877 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -127,20 +127,14 @@ The decision to use /** to start a doc-comment ensures a unique distinction from **The placment describes the relationship between doc-comments and the expression they are documenting.** -- Doc-comments are placed before the documentable node. Only `WHITESPACES` are allowed in between. +- Doc-comments are placed before the documentable node. Only `WHITESPACES` are allowed in between. ([Examples](#basic-examples)) - `WHITESPACES` are: `[\n \r ' ' \t]`. - - Example: [Basic examples](#basic-examples) -- The documentation present before the `attribute path` describes the body of the attribute. -- Example: [Attributes](#Attributes) - - In case placement is ambiguous, the one closer to the body has higher precedence. - - Example: [Ambiguous placement](#ambiguous-placement) - - -- All partial functions of a curried lambda can share the same placement with the outermost lambda. - - Example: [partial lambda functions](#partial-lambda-functions) +- The documentation present before the `attribute path` describes the body of the attribute. ([Examples](#Attributes)) + - In case placement is ambiguous, the one closer to the body has higher precedence. ([Examples](#ambiguous-placement)) + +- All partial functions of a curried lambda can share the same placement with the outermost lambda. ([Examples](#partial-lambda-functions)) - > Note: Research of the RFC Sheperds Team showed that this allows for intuitive placements like are already done in nixpkgs. ### Examples @@ -341,19 +335,19 @@ See future work. ## All considered outer formats -| Property / Approach | `##` | `/** */` | `Javadoc` | `/*\|` or `/*^` | -|---|---|---|---|---| -| Inspired by | Rust | Current nixpkgs.lib | C++/Java/Javascript | Haskell Haddock | -| Changes the existing code by | Much | Less | Even More | Less | -| Needs Termination | No | Yes | Yes | Yes | -| Indentation | Clear | like Nix's multiline strings, thus **Intuitive** | Clear | ? | -| Needs vertical space | No | Yes | Yes | Yes | -| Visual distinction from comments | High | Low | Medium | Medium | -| Needs Autocompletion (Language Server) to continue the next line. | Yes | No | Yes | No | -| Punctuation Variations / Amount of different special characters | 1 (Less) | 2 (Medium) | 2 (Medium) | 3 (More) | -| Markdown compatibility (also depends on indentation clarity) | Good, but visual conflicts with headings `# Title` | Good | Medium | Good | -| breaks when interrupted with newlines | Yes | No | ? | No | -| Simplicity (Brainload) | Medium | Simple | Complex | More Complex | +| Property / Approach | `##` | `/** */` | `Javadoc` | `/*\|` or `/*^` | All comments are doc-comments | +|---|---|---|---|---|---| +| Inspired by | Rust | Current nixpkgs.lib | C++/Java/Javascript | Haskell Haddock | Current nixpkgs.lib | +| Changes the existing code by | Much | Less | Even More | Less | No change | +| Needs Termination | No | Yes | Yes | Yes | ? | +| Indentation | Clear | like Nix's multiline strings, thus **Intuitive** | Clear | ? | ? | +| Needs vertical space | No | Yes | Yes | Yes | ? | +| Visual distinction from comments | High | Low | Medium | Medium | No distinction at all | +| Needs Autocompletion (Language Server) to continue the next line. | Yes | No | Yes | No | ? | +| Punctuation Variations / Amount of different special characters | 1 (Less) | 2 (Medium) | 2 (Medium) | 3 (More) | None | +| Markdown compatibility (also depends on indentation clarity) | **Visual conflicts with headings `# Title`** | Good | Medium | Good | Only with multiline comments | +| breaks when interrupted with newlines | Yes | No | ? | No | | +| Simplicity (Brainload) | Medium | Simple | Complex | More Complex | | ### Refactoring note @@ -375,48 +369,29 @@ While this allows the most freedom, it is usually considered the best option, no - [RFC72](https://github.com/NixOS/rfcs/blob/master/rfcs/0072-commonmark-docs.md) defines commonMark as the official documentation format. This is why we decided to follow this convention. Writing plain text is still possible. -## Consequences of not implementing this - -- By not implementing this feature, Nix gains no ability for tool-generated documentation. -- Many existing comments written for documentation will remain imperceptible for automated tooling. - # Unresolved questions [unresolved]: #unresolved-questions -- Will `nix` itself implement native support like in rust -> `cargo doc`? +- Migration path for nixpkgs comments. +- How to document the `arguments`. Should there be some markdown equivalent to `@param` i.e. `# Params`? + - Possible answer: Future RFC might need to clarify that. # Future work [Future]: #future-work -## Migrate the existing comments - -Reformatting existing doc-comments in Nixpkgs, but also filtering out false-positives, i.e. those that should not be part of the API documentation. - -## nixpkgs Manual tooling +## Migrate existing nixpkgs comments -The current tooling needs to be adopted to this change. With supporting the new format the currently existing scope can be retained to build the nixpkgs manual. +Reformatting existing doc-comments in nixpkgs. -## Enhanced Documentation generators +## nixpkgs Manual -We think that a future documentation tool could be out of one of the two following categories. - -- (1) Tools that utilize static code analysis and configuration files. - -- (2) Tools that use dynamic evaluation to attach name and value relationships and provides more accurate documentation with less configuration overhead. - -For the beginning it seems promising to start with the static approach (1). In the long term a dynamic approach (2) seems more promising and accurate but requires a much deeper understanding of compilers and evaluators. - -However the border between those two variants is not strict and we might find future tools that fit our needs just perfectly. +The current nixpkgs manual needs to be adopted to this change. ## Native support in Nix -- `NixOS/nix` should implement native support for doc-comments so that our users do not have to rely on nixpkgs or external tools. Those tools can still exist and provide more custom functionality, but documenting nix expressions should be natively possible. - -We propose to add: +- `NixOS/nix` should implement native support for doc-comments. -- Adding two builtins: `getAttrDoc` and `getLambdaDoc` -- both introduced with `unsafe` prefix -- until properly stabilized according to the official process for feature stabilization. +> Note: We considered implementation details, but specifying those is out of scope for this rfc. ## References From b986270b918e10b82912bb4be918938a6f302faf Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Fri, 29 Sep 2023 11:08:57 +0200 Subject: [PATCH 081/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 3bc3fb877..eec616e32 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -117,8 +117,6 @@ In the following we give a comprehensive overview to our decision that we've mad Adopting CommonMark as the content for all doc-comments brings the benefits of widely accepted and understood documentation format in tech projects, while maintaining profitability and consistency within the Nix ecosystem by aligning with existing [NixOS/RFC-72](https://github.com/NixOS/rfcs/blob/master/rfcs/0072-commonmark-docs.md). -The content indentation inherits the indentation logic from nixs' multiline strings `'' ''` allowing any arbitrary indentation the user might prefer. - ## `/** */` is the doc-comment format The decision to use /** to start a doc-comment ensures a unique distinction from regular comments while still allowing seamless writing without IDE or editor support. This choice not only provides the best developer experience but also minimizes the need for additional tooling overhead. @@ -393,6 +391,13 @@ The current nixpkgs manual needs to be adopted to this change. > Note: We considered implementation details, but specifying those is out of scope for this rfc. +Related POCs: + +- https://github.com/NixOS/nix/pull/9054 +- https://github.com/NixOS/nix/pull/1652 +- https://github.com/NixOS/nix/pull/5527 +- https://github.com/NixOS/nix/commit/8e252320f52426c734883709eb398d2161c3fe82 + ## References ### Other Conventions From b133e657af7225c80eea3c47e7573474cf14d794 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Fri, 29 Sep 2023 11:12:04 +0200 Subject: [PATCH 082/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index eec616e32..90a9923b6 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -336,7 +336,7 @@ See future work. | Property / Approach | `##` | `/** */` | `Javadoc` | `/*\|` or `/*^` | All comments are doc-comments | |---|---|---|---|---|---| | Inspired by | Rust | Current nixpkgs.lib | C++/Java/Javascript | Haskell Haddock | Current nixpkgs.lib | -| Changes the existing code by | Much | Less | Even More | Less | No change | +| Changes the existing code by | Much | Less | Even More | Less | ? | | Needs Termination | No | Yes | Yes | Yes | ? | | Indentation | Clear | like Nix's multiline strings, thus **Intuitive** | Clear | ? | ? | | Needs vertical space | No | Yes | Yes | Yes | ? | From f415a11688843e6fffa88bbb367b959aacaacbd6 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Fri, 29 Sep 2023 13:40:53 +0200 Subject: [PATCH 083/110] Add non goal --- rfcs/0145-doc-strings.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 90a9923b6..384bef402 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -49,6 +49,8 @@ The following are the envisioned goals. - Discuss in which tool doc-comments are parsed and rendered. This could be an external tool, or Nix, or something else entirely, but that's out of scope for this RFC. +- Implementation details are not specified. The RFC shepherd group has some (feature incomplete) POC's sufficient for a generic specification. (See [Native support in Nix](#Native-support-in-Nix) ) + ## Current State A third-party tool called [nixdoc](https://github.com/nix-community/nixdoc) has emerged, which codifies its own rules as to the internal and external formats of a Nix doc-comment. This tool has seen some adoption, notably for the `nixpkgs.lib` functions. From 6864009daaa3afa64b30acfceda96057ba125c22 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 26 Oct 2023 13:33:24 +0200 Subject: [PATCH 084/110] remove unnecessary sentence Co-authored-by: Lassulus --- rfcs/0145-doc-strings.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 384bef402..e86d7c59e 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -18,8 +18,6 @@ This RFC includes two concerns that define a doc-comment: - Outer format rules to allow distinction between regular comments and doc-comments - Inner format rules that describe the required format of a doc-comment. -However, both concerns relate closely to each other; It makes sense and reduces bureaucracy to address both in a single RFC. - # Definitions For this RFC, we adopt the following definitions: From 174040a7725874304e77df75971f5a51a9e1d49c Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 26 Oct 2023 13:34:20 +0200 Subject: [PATCH 085/110] Code has interface Co-authored-by: Robert Hensing --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index e86d7c59e..6d5404c73 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -22,7 +22,7 @@ This RFC includes two concerns that define a doc-comment: For this RFC, we adopt the following definitions: -- **doc-comment**: A structured comment documenting the code's API. +- **doc-comment**: A structured comment documenting the code's interface. - **internal comment**: A free-form comment for those interested in the code's implementation. The doc-comment properties are grouped into these subcategories: From b6bea69299193e37983cd55b320518fbd30a99ed Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 26 Oct 2023 13:35:53 +0200 Subject: [PATCH 086/110] More specific outer format Co-authored-by: Robert Hensing --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 6d5404c73..f2e59317c 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -27,7 +27,7 @@ For this RFC, we adopt the following definitions: The doc-comment properties are grouped into these subcategories: -- **Outer format**: Specifies rules linking code (API) and doc-comments. (e.g. placement, syntax rules) +- **Outer format**: Specifies rules linking code and doc-comments, regarding placement within expressions and the chosen lexical syntax of the comment within the existing Nix language. - **Inner format**: Specifies rules affecting the comment's actual content. (e.g. usage of commonMark) From 70bca5e479b038ab58e7df6796f280f9a14a56ca Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 26 Oct 2023 13:36:41 +0200 Subject: [PATCH 087/110] Single format simplifies tooling. Co-authored-by: Robert Hensing --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index f2e59317c..26a3af2ef 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -119,7 +119,7 @@ Adopting CommonMark as the content for all doc-comments brings the benefits of w ## `/** */` is the doc-comment format -The decision to use /** to start a doc-comment ensures a unique distinction from regular comments while still allowing seamless writing without IDE or editor support. This choice not only provides the best developer experience but also minimizes the need for additional tooling overhead. +The decision to use /** to start a doc-comment ensures a unique distinction from regular comments while still allowing seamless writing without IDE or editor support. A single choice provides the best developer experience and simplifies tooling. ## Placement From 72359a2d3d7e44fb0529c69730278df41bf15602 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 26 Oct 2023 13:37:00 +0200 Subject: [PATCH 088/110] fix typo Co-authored-by: Robert Hensing --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 26a3af2ef..90d680146 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -123,7 +123,7 @@ The decision to use /** to start a doc-comment ensures a unique distinction from ## Placement -**The placment describes the relationship between doc-comments and the expression they are documenting.** +**The placement describes the relationship between doc-comments and the expression they are documenting.** - Doc-comments are placed before the documentable node. Only `WHITESPACES` are allowed in between. ([Examples](#basic-examples)) - `WHITESPACES` are: `[\n \r ' ' \t]`. From 00e76a7a4fe241431bb2320572262375c17d6c63 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Fri, 27 Oct 2023 16:49:30 +0100 Subject: [PATCH 089/110] Typo Co-authored-by: asymmetric --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 90d680146..5b0ba15de 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -415,7 +415,7 @@ Related POCs: ## Related discussions - - https://discourse.nixos.org/t/2023-04-13-documentation-team-meeting-notes-41/27264) +- https://discourse.nixos.org/t/2023-04-13-documentation-team-meeting-notes-41/27264 - https://github.com/NixOS/nix/issues/3904 - https://github.com/NixOS/nix/issues/228 From d82915754b537d5e764ec3f070b1c322013477e8 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Fri, 27 Oct 2023 16:51:48 +0100 Subject: [PATCH 090/110] Correct statement about old pr Co-authored-by: Robert Hensing --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 5b0ba15de..0e0fe6537 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -424,4 +424,4 @@ Related POCs: - https://github.com/NixOS/nix/pull/5527 - https://github.com/NixOS/nix/pull/1652 -[#1652](https://github.com/NixOS/nix/pull/1652) gets closest what this RFC imagines, implementation is almost done. It certainly has some limitations, but got stuck on politics and diverging opinions about markdown and syntax rules. +[#1652](https://github.com/NixOS/nix/pull/1652) gets closest what this RFC imagines, implementation is almost done. It certainly has some limitations, but was not merged due to uncertainty and complexity. From 1c1eaccc2311987cefe87d78168db41407966459 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Fri, 27 Oct 2023 16:53:27 +0100 Subject: [PATCH 091/110] Dont overspecify whitespaces Co-authored-by: Robert Hensing --- rfcs/0145-doc-strings.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 0e0fe6537..abe1f6f87 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -125,8 +125,7 @@ The decision to use /** to start a doc-comment ensures a unique distinction from **The placement describes the relationship between doc-comments and the expression they are documenting.** -- Doc-comments are placed before the documentable node. Only `WHITESPACES` are allowed in between. ([Examples](#basic-examples)) - - `WHITESPACES` are: `[\n \r ' ' \t]`. +- Doc-comments are placed before the documentable node. Only whitespace is allowed in between. ([Examples](#basic-examples)) - The documentation present before the `attribute path` describes the body of the attribute. ([Examples](#Attributes)) - In case placement is ambiguous, the one closer to the body has higher precedence. ([Examples](#ambiguous-placement)) From 74820d50bf154de6c5e403393d2fb3939bff13a6 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Fri, 27 Oct 2023 17:02:11 +0100 Subject: [PATCH 092/110] Update rfcs/0145-doc-strings.md Co-authored-by: asymmetric --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index abe1f6f87..03d5a32ad 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -97,7 +97,7 @@ The lack of a formal definition of a doc-comment also means there is no reliable ### References to the problems above -> This curated link collection highlights the Nix ecosystem's inconsistencies, a primary focus of this RFC. +This curated link collection highlights the Nix ecosystem's inconsistencies, a primary focus of this RFC. #### nixpkgs - comment examples From fedeadfd925ec22dc6ba8b6166eb81ceed6c284d Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sat, 28 Oct 2023 16:43:29 +0200 Subject: [PATCH 093/110] strictly follow rfc72 Co-authored-by: Silvan Mosberger --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 03d5a32ad..7ded6b212 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -113,7 +113,7 @@ In the following we give a comprehensive overview to our decision that we've mad ## CommonMark -**CommonMark is the content of all doc-comments.** +**CommonMark according to [RFC 72](https://github.com/NixOS/rfcs/blob/master/rfcs/0072-commonmark-docs.md) is the content of all doc-comments.** Adopting CommonMark as the content for all doc-comments brings the benefits of widely accepted and understood documentation format in tech projects, while maintaining profitability and consistency within the Nix ecosystem by aligning with existing [NixOS/RFC-72](https://github.com/NixOS/rfcs/blob/master/rfcs/0072-commonmark-docs.md). From 4ee56365b2ef0194ec7ab0bd2fabb732825c68ed Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 29 Oct 2023 14:04:44 +0100 Subject: [PATCH 094/110] Attribute path leafs: add example --- rfcs/0145-doc-strings.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 7ded6b212..ec449a9b3 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -170,6 +170,18 @@ assigned = x: x; } ``` +### Attribute path + +```nix +{ + /** Doc 1 bound to 'c' */ + ↓ ↓ + a.b.c = 1; +} + +# Documents only the expression bound to 'c'. NOT the attribute set bound to 'a' or 'b'. +``` + #### Ambiguous placement ```nix From cc333bd35bff922ad240700ee2a14f933a9bb4df Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 29 Oct 2023 14:15:23 +0100 Subject: [PATCH 095/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index ec449a9b3..f93b31cc8 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -23,7 +23,7 @@ This RFC includes two concerns that define a doc-comment: For this RFC, we adopt the following definitions: - **doc-comment**: A structured comment documenting the code's interface. -- **internal comment**: A free-form comment for those interested in the code's implementation. +- **comment**: A free-form comment for to help with understanding the implementation and design concerns that aren't obvious. The doc-comment properties are grouped into these subcategories: @@ -39,7 +39,7 @@ The following are the envisioned goals. - Create distinct outer and inner formats for Nix doc-comments to enable accurate automated parsing and extraction. - Ensure clear differentiation from internal comments, making them accessible to tooling solutions such as documentation rendering and building. -- Migrate `nixpkgs'` comments to this format. +- Convert `nixpkgs` comments intended for documentation into this format. - In addition, the developer experience and adherence to established conventions should be taken into account. Equally important is ensuring that doc comments remain effortless to compose and comprehend, though it is essential to acknowledge that this aspect may vary subjectively based on personal preferences. @@ -125,14 +125,14 @@ The decision to use /** to start a doc-comment ensures a unique distinction from **The placement describes the relationship between doc-comments and the expression they are documenting.** +The following rules apply in descending order of precedence: + - Doc-comments are placed before the documentable node. Only whitespace is allowed in between. ([Examples](#basic-examples)) - The documentation present before the `attribute path` describes the body of the attribute. ([Examples](#Attributes)) - In case placement is ambiguous, the one closer to the body has higher precedence. ([Examples](#ambiguous-placement)) -- All partial functions of a curried lambda can share the same placement with the outermost lambda. ([Examples](#partial-lambda-functions)) - -> Note: Research of the RFC Sheperds Team showed that this allows for intuitive placements like are already done in nixpkgs. +- All partial functions of a curried lambda can share the same placement with the outermost lambda. ([Examples](#partial-lambda-functions)) ### Examples From b26faa556839fdf97958350c4e41c0dd87b9ae4e Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Mon, 30 Oct 2023 16:12:43 +0100 Subject: [PATCH 096/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 50 +++++++++++++++++++++++++--------------- 1 file changed, 32 insertions(+), 18 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index f93b31cc8..28c2eeffb 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -29,11 +29,20 @@ The doc-comment properties are grouped into these subcategories: - **Outer format**: Specifies rules linking code and doc-comments, regarding placement within expressions and the chosen lexical syntax of the comment within the existing Nix language. -- **Inner format**: Specifies rules affecting the comment's actual content. (e.g. usage of commonMark) +- **Inner format**: Specifies rules affecting the comment's actual content. Utilizing content formatting within doc-comments ensures consistent rendering, akin to those achieved with CommonMark. # Motivation [motivation]: #motivation +The primary motivation behind doc-comments is to provide documentation for functions, types, modules, and other elements of the codebase. Good documentation is essential for understanding how to use a library or module correctly. Doc-comments allow developers to provide explanations, examples, and usage guidelines directly alongside the code, making it easier for others (and themselves) to understand and use the code effectively. + +Many development tools and IDEs (Integrated Development Environments) can parse doc-comments and provide features like autocompletion, hover tooltips, and documentation pop-ups. This tooling support enhances developer productivity by making it easier to explore and use functions and modules without referring to external documentation. + +Writing doc-comments also encourages developers to think about the clarity and correctness of their code. By documenting functions and modules, developers are more likely to write clean, self-explanatory code, which can lead to better code quality and maintainability. + +Overall doc-comments can serve as a vital tool for documentation, code understanding, tooling support, onboarding for new contributors and code quality. + +# Goals The following are the envisioned goals. - Create distinct outer and inner formats for Nix doc-comments to enable accurate automated parsing and extraction. @@ -43,9 +52,9 @@ The following are the envisioned goals. - In addition, the developer experience and adherence to established conventions should be taken into account. Equally important is ensuring that doc comments remain effortless to compose and comprehend, though it is essential to acknowledge that this aspect may vary subjectively based on personal preferences. -# Non-goals +## Non-goals -- Discuss in which tool doc-comments are parsed and rendered. This could be an external tool, or Nix, or something else entirely, but that's out of scope for this RFC. +- Discuss in which tool doc-comments are parsed and rendered. This could be an external tool, native nix, or something else entirely, but that's out of scope for this RFC. - Implementation details are not specified. The RFC shepherd group has some (feature incomplete) POC's sufficient for a generic specification. (See [Native support in Nix](#Native-support-in-Nix) ) @@ -93,7 +102,7 @@ Among the formats encountered in the wild, the one used in `nixpkgs/lib` is the ### Impossible to differentiate from internal comments -The lack of a formal definition of a doc-comment also means there is no reliable way to distinguish them from internal comments, which would result in automatically-produced API documentation which includes the wrong type of comments. +The lack of a formal definition of a doc-comment also means there is no reliable way to distinguish them from internal comments, which makes it impossible to access from doc-comments from tooling. Futhermore it makes it very hard to generate accurate and complete reference documentation. ### References to the problems above @@ -101,10 +110,10 @@ This curated link collection highlights the Nix ecosystem's inconsistencies, a p #### nixpkgs - comment examples -- [lib/attrsets](https://github.com/NixOS/nixpkgs/blob/master/lib/attrsets.nix) -- [trivial-builders](https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/trivial-builders.nix) -- [stdenv/mkDerivation](https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix) -- [nixos/lib/make-disk-image](https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-disk-image.nix) +- [lib/attrsets](https://github.com/NixOS/nixpkgs/blob/5323fbf70331f8a7c47f1b4f49841cf74507f77f/lib/attrsets.nix) +- [trivial-builders](https://github.com/NixOS/nixpkgs/blob/5323fbf70331f8a7c47f1b4f49841cf74507f77f/pkgs/build-support/trivial-builders/default.nix) +- [stdenv/mkDerivation](https://github.com/NixOS/nixpkgs/blob/5323fbf70331f8a7c47f1b4f49841cf74507f77f/pkgs/stdenv/generic/make-derivation.nix) +- [nixos/lib/make-disk-image](https://github.com/NixOS/nixpkgs/blob/5323fbf70331f8a7c47f1b4f49841cf74507f77f/nixos/lib/make-disk-image.nix) # Design [design]: #detailed-design @@ -223,11 +232,11 @@ Each subsection here contains a decision along with arguments and counter-argume ## `/**` to start a doc-comment -**Observing**: Doc-comments' outer format should be a distinctive subset of regular comments. Nevertheless, it should allow native writing without an IDE or editor support. +**Observing**: The use of `/**` to initiate a doc-comment is a widely accepted convention in many programming languages. It indicates the beginning of a comment block specifically meant for documentation purposes. -**Considering**: `/** {content} */` where `{content}` is the inner format. +**Considering**: Doc-comments' outer format should be a distinctive subset of regular comments. Nevertheless, it should allow native writing without an IDE or editor support. -**Decision**: use `/** {content} */` as the outer format. +**Decision**: `/** {content} */` where `/**` is used to start the doc-comment. `Example` @@ -274,9 +283,9 @@ Each subsection here contains a decision along with arguments and counter-argume ## CommonMark as the content of doc-comments -**Observing**: Doc-comments' content should be intuitive to read and write and straightforward to render. The nixdoc convention is only widely adopted in certain places (e.g., /lib) in nixpkgs. It may also come from a need for more understanding of the current self-cooked format. +**Observing**: The use of CommonMark, a widely recognized and standardized format for documents, is prevalent in the documentation of code and software libraries. -**Considering**: CommonMark as the content format. +**Considering**: Doc-comments' content should be intuitive to read and write and straightforward to render. Furthermore it should follow established conventions in the nix ecosystem. **Decision**: CommonMark is the content of all doc-comments. @@ -285,7 +294,7 @@ Each subsection here contains a decision along with arguments and counter-argume
Arguments -- (+) CommonMark is the official format in nix; Decided in [RFC72](https://github.com/NixOS/rfcs/blob/master/rfcs/0072-commonmark-docs.md). +- (+) CommonMark is the official format for Nix documentation; Decided in [RFC72](https://github.com/NixOS/rfcs/blob/master/rfcs/0072-commonmark-docs.md). - (+) It Would be consistent if this RFC builds upon the previous one. - (+) Further Arguments for CommonMark, in general, can be found in RFC72 - (+) Allows copy-paste from and to markdown files. We allow easy refactoring if documentation arises and needs to be split into separate files. @@ -381,9 +390,8 @@ This is why we decided to follow this convention. Writing plain text is still po # Unresolved questions [unresolved]: #unresolved-questions -- Migration path for nixpkgs comments. -- How to document the `arguments`. Should there be some markdown equivalent to `@param` i.e. `# Params`? - - Possible answer: Future RFC might need to clarify that. +- Migration path for nixpkgs comments. +- How to document the `arguments`. Should there be some markdown equivalent to `@param` i.e. `# Params`? This RFC intentionally leaves this question unanswered, allowing for further discussion and decision-making in the future. # Future work [Future]: #future-work @@ -392,7 +400,11 @@ This is why we decided to follow this convention. Writing plain text is still po Reformatting existing doc-comments in nixpkgs. -## nixpkgs Manual +## Tooling + +All current rendering tooling solutions should support displaying the specified doc-comment format. + +### nixpkgs Manual renderer The current nixpkgs manual needs to be adopted to this change. @@ -400,6 +412,8 @@ The current nixpkgs manual needs to be adopted to this change. - `NixOS/nix` should implement native support for doc-comments. +Primarily because the *Nix* language requires access to the actual evaluator for building correct relations between doc-comment and and the expressions they document. + > Note: We considered implementation details, but specifying those is out of scope for this rfc. Related POCs: From d6c259002ad0a3656f899639e4dd71f4ab243a02 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Tue, 31 Oct 2023 16:13:32 +0100 Subject: [PATCH 097/110] Specify documentable nodes. Not only expressions --- rfcs/0145-doc-strings.md | 61 +++++++++++++++++++++++++++++++++++----- 1 file changed, 54 insertions(+), 7 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 28c2eeffb..d509b4c01 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -132,16 +132,22 @@ The decision to use /** to start a doc-comment ensures a unique distinction from ## Placement -**The placement describes the relationship between doc-comments and the expression they are documenting.** +**The placement describes the relationship between doc-comments and the documentable node.** + +A **documentable node** can be: + +- Expression +- Binding +- Lambda Formal The following rules apply in descending order of precedence: -- Doc-comments are placed before the documentable node. Only whitespace is allowed in between. ([Examples](#basic-examples)) +- Doc-comments are placed before the **documentable node**. Only whitespace is allowed in between. ([Examples](#basic-examples)) - The documentation present before the `attribute path` describes the body of the attribute. ([Examples](#Attributes)) - In case placement is ambiguous, the one closer to the body has higher precedence. ([Examples](#ambiguous-placement)) -- All partial functions of a curried lambda can share the same placement with the outermost lambda. ([Examples](#partial-lambda-functions)) +- All partial functions of a curried lambda can share the same placement with the outermost lambda. ([Examples](#partial-lambda-functions)) ### Examples @@ -211,17 +217,58 @@ int = /**Doc A*/1; # Dynamic attribute ``` +#### `Let .. in ..` binding + +```nix +let + /** Documentation for the id function*/ + ↓ + a = x: x; +in + a + +# Documentation can still be retrieved. +``` + #### Partial lambda functions ```nix /**Generic doc for all partial lambda functions*/ ↓1 ↓2 ↓3 -x: ({y, ...}: z: x + y) * z; +x: ({y, ...}: z: x + y * z) # All partial functions can share the same placement with the outermost lambda. -# 1 -> Lambda Value `x: ({y, ...}: z: x + y) * z` -# 2 -> Lambda Value `({y, ...}: z: x + y) * z` -# 3 -> Lambda Value `z: x + y) * z` +# 1 -> Lambda Value `x: ({y, ...}: z: x + y * z)` +# 2 -> Lambda Value `({y, ...}: z: x + y * z)` +# 3 -> Lambda Value `z: x + y * z)` +``` + +> Allows for fallback documentation in case documentation on the partially applied function cannot be written or retrieved. + +#### Lambda formals + +```nix +/**Doc for the whole lambda function*/ +{ + /**Doc for attribute 'a'*/ + a +}: + a +``` + +```nix +/**Doc for the whole lambda function*/ +{ + /**Doc for attribute 'a'*/ + a, + ... +}@args: + a + args.b; + +# `...` and `@args` cannot (yet) be documented. +# We recommend: +# - Include all known arguments in the pattern `{ }:` +# - Document how more arguments can be passed to the function in `Doc for the whole lambda function` ``` # Decisions From fcc24cdf472a22176aa0ada6e656678c4c003192 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Tue, 31 Oct 2023 16:18:40 +0100 Subject: [PATCH 098/110] Update 0145-doc-strings.md --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index d509b4c01..26bfb77f0 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -459,7 +459,7 @@ The current nixpkgs manual needs to be adopted to this change. - `NixOS/nix` should implement native support for doc-comments. -Primarily because the *Nix* language requires access to the actual evaluator for building correct relations between doc-comment and and the expressions they document. +Primarily because the *Nix* language requires access to the actual evaluator for building correct relations between doc-comment and and the node they document. > Note: We considered implementation details, but specifying those is out of scope for this rfc. From 51f5bcc21f2c05f6fc3a9715e9d3a7ec5c9ef743 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Fri, 3 Nov 2023 15:24:42 +0100 Subject: [PATCH 099/110] typo: partial applications Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com> --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 26bfb77f0..a3c44f5f7 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -147,7 +147,7 @@ The following rules apply in descending order of precedence: - The documentation present before the `attribute path` describes the body of the attribute. ([Examples](#Attributes)) - In case placement is ambiguous, the one closer to the body has higher precedence. ([Examples](#ambiguous-placement)) -- All partial functions of a curried lambda can share the same placement with the outermost lambda. ([Examples](#partial-lambda-functions)) +- All partial applications of a curried lambda can share the same placement with the outermost lambda. ([Examples](#partial-lambda-functions)) ### Examples From 1e3f041fe0e480832115a93c6b541999496e7073 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sat, 4 Nov 2023 10:29:43 +0100 Subject: [PATCH 100/110] feedback from comment --- rfcs/0145-doc-strings.md | 81 +++++++++++++++++++++++----------------- 1 file changed, 47 insertions(+), 34 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index a3c44f5f7..cb62afff0 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -142,7 +142,7 @@ A **documentable node** can be: The following rules apply in descending order of precedence: -- Doc-comments are placed before the **documentable node**. Only whitespace is allowed in between. ([Examples](#basic-examples)) +- Doc-comments are placed before the **documentable node**. Only whitespace or non-doc comments are allowed in between. ([Examples](#basic-examples)) - The documentation present before the `attribute path` describes the body of the attribute. ([Examples](#Attributes)) - In case placement is ambiguous, the one closer to the body has higher precedence. ([Examples](#ambiguous-placement)) @@ -250,27 +250,12 @@ x: ({y, ...}: z: x + y * z) ```nix /**Doc for the whole lambda function*/ { - /**Doc for attribute 'a'*/ + /**Doc for formal 'a'*/ a }: a ``` -```nix -/**Doc for the whole lambda function*/ -{ - /**Doc for attribute 'a'*/ - a, - ... -}@args: - a + args.b; - -# `...` and `@args` cannot (yet) be documented. -# We recommend: -# - Include all known arguments in the pattern `{ }:` -# - Document how more arguments can be passed to the function in `Doc for the whole lambda function` -``` - # Decisions Each subsection here contains a decision along with arguments and counter-arguments for (+) and against (-) that decision. @@ -352,29 +337,29 @@ Each subsection here contains a decision along with arguments and counter-argume
-## Single-line doc-comments (do not exist) +## Single-line doc-comments (not specified by this rfc) **Observing**: Nix offers two variants of comments; single- (`#`) and multi-line comments (`/* */`). There may be use cases where it is desirable to have a form of single-line comments subtyped for doc-comment purposes. -**Considering**: Single-line comment for documentation. (Starting a doc-comment with `##`) +**Considering**: Single-line comment for documentation. (Starting a doc-comment with `##` or similar) -**Decision**: Single-line comments (starting with `##`) **cannot be used** in any form for documentation puposes. +**Decision**: Single-line comments (starting with `##`) **remain unspecified** by this rfc.
Arguments -- (+) It Would be consistent with providing variants for both nix comments. -- (-) Doc-comments should have only one variant to reduce complexity. -- (-) documentation will likely everytime take up more than one line. -- (-) If documentation grows bigger than one line, refactoring into a multiline-doc-comment must occur. -- (+) Offer the choice. +- (-) It Would be consistent with providing variants for both nix comments. +- (+) Doc-comments have only one variant to reduce complexity. +- (+) documentation will likely everytime take up more than one line. +- (+) If documentation grows bigger than one line, refactoring into a multiline-doc-comment must occur. +- (-) Offer the choice. - (o) Single lines could also be concatenated to form multi-line documentation. - - (+) Convinience - - (-) Technical more complex -- (+) Takes up less vertical space -- (-) Visually confusing when every line starts with a `#` character. - - (-) Potential visual conflicts with the content that is markdown (e.g. with headings). -- (+) Indentation of the content is clear. + - (-) Convinience + - (+) Technical more complex +- (-) Takes up less vertical space +- (+) Visually confusing when every line starts with a `#` character. + - (+) Potential visual conflicts with the content that is markdown (e.g. with headings). +- (-) Indentation of the content is clear.
@@ -383,15 +368,31 @@ Each subsection here contains a decision along with arguments and counter-argume ## Changes the existing comments inside the code base -This could be automated. +This could be (partially) automated. (see our [codemod](https://github.com/nix-community/docnix/tree/3c0531cb5b4c9f3e9069b73d19e6c4be8508d905/codemod) ) Also, the migration could be performed piecemal, starting perhaps with `nixpkgs.lib`. See future work. -## Tooling to produce the nixpkgs manual +### Migration Example -Change or write a new tool used to produce Nixpkgs library function documentation. Because this is a breaking change. +TODO: + +`old format` +```nix + +``` + +-> + +`new format` +```nix + +``` + +## Breaking the nixpkgs manual + +This is a breaking change, to the current nixpkgs manual tooling Nixpkgs library function documentation. See future work. @@ -447,14 +448,26 @@ This is why we decided to follow this convention. Writing plain text is still po Reformatting existing doc-comments in nixpkgs. +Action points: + +- [ ] Change comments to markdown. +- [ ] Migrate nixdoc 'argument documentation' format. + ## Tooling All current rendering tooling solutions should support displaying the specified doc-comment format. +Currently at least: + ### nixpkgs Manual renderer The current nixpkgs manual needs to be adopted to this change. +We expect changes in the following to be neccessary: + +- [nixos_render_docs](https://github.com/NixOS/nixpkgs/tree/e4082efedb483eb0478c3f014fa851449bca43f9/pkgs/tools/nix/nixos-render-docs/src) +- [nixdoc](https://github.com/nix-community/nixdoc) + ## Native support in Nix - `NixOS/nix` should implement native support for doc-comments. From 6471c42ced58ebebcb63ce19ec627a3d30693286 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Wed, 8 Nov 2023 14:55:06 +0100 Subject: [PATCH 101/110] Add migration example --- rfcs/0145-doc-strings.md | 74 +++++++++++++++++++++++++++++++--------- 1 file changed, 58 insertions(+), 16 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index cb62afff0..f61ea5455 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -374,22 +374,6 @@ Also, the migration could be performed piecemal, starting perhaps with `nixpkgs. See future work. -### Migration Example - -TODO: - -`old format` -```nix - -``` - --> - -`new format` -```nix - -``` - ## Breaking the nixpkgs manual This is a breaking change, to the current nixpkgs manual tooling Nixpkgs library function documentation. @@ -453,6 +437,64 @@ Action points: - [ ] Change comments to markdown. - [ ] Migrate nixdoc 'argument documentation' format. +### Migration Example + +The following shows one of the many possible ways to migrate the current `nixpkgs.lib` comments. + +> Refactoring Note: the current `nixdoc` feature 'Function arguments' should be changed or removed. Doc-comments that relate to a lambda use this placement already. + +`lib/attrsets.nix (old format)` +````nix +/* Filter an attribute set by removing all attributes for which the + given predicate return false. + + Example: + filterAttrs (n: v: n == "foo") { foo = 1; bar = 2; } + => { foo = 1; } + + Type: + filterAttrs :: (String -> Any -> Bool) -> AttrSet -> AttrSet +*/ +filterAttrs = + # Predicate taking an attribute name and an attribute value, which returns `true` to include the attribute, or `false` to exclude the attribute. + pred: + # The attribute set to filter + set: + listToAttrs (concatMap (name: let v = set.${name}; in if pred name v then [(nameValuePair name v)] else []) (attrNames set)); +```` + +-> + +`lib/attrsets.nix (new format)` +````nix +/** + Filter an attribute set by removing all attributes for which the + given predicate return false. + + # Example + + ```nix + filterAttrs (n: v: n == "foo") { foo = 1; bar = 2; } + => { foo = 1; } + ``` + + # Type + + ``` + filterAttrs :: (String -> Any -> Bool) -> AttrSet -> AttrSet + ``` + + # Arguments + + - [pred] Predicate taking an attribute name and an attribute value, which returns `true` to include the attribute, or `false` to exclude the attribute. + - [set] The attribute set to filter +*/ +filterAttrs = + pred: + set: + listToAttrs (concatMap (name: let v = set.${name}; in if pred name v then [(nameValuePair name v)] else []) (attrNames set)); +```` + ## Tooling All current rendering tooling solutions should support displaying the specified doc-comment format. From e6870a0e6f6d362c47f1f1ee1f6c1fa76336b7af Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 9 Nov 2023 09:59:46 +0100 Subject: [PATCH 102/110] Add link to codemod on migration example --- rfcs/0145-doc-strings.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index f61ea5455..e9cdeee85 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -441,6 +441,8 @@ Action points: The following shows one of the many possible ways to migrate the current `nixpkgs.lib` comments. +> We managed to partially automate this effort with a [codemod](https://github.com/nix-community/docnix/tree/3c0531cb5b4c9f3e9069b73d19e6c4be8508d905/codemod) + > Refactoring Note: the current `nixdoc` feature 'Function arguments' should be changed or removed. Doc-comments that relate to a lambda use this placement already. `lib/attrsets.nix (old format)` From 9dcad4abae62b676c3340ec303f8e843d1b67477 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 9 Nov 2023 11:08:21 +0100 Subject: [PATCH 103/110] Add non goal: Styling of markdown --- rfcs/0145-doc-strings.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index e9cdeee85..89800ff48 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -56,6 +56,8 @@ The following are the envisioned goals. - Discuss in which tool doc-comments are parsed and rendered. This could be an external tool, native nix, or something else entirely, but that's out of scope for this RFC. +- How to style or format the commonmark inside of a doc-comment. + - Implementation details are not specified. The RFC shepherd group has some (feature incomplete) POC's sufficient for a generic specification. (See [Native support in Nix](#Native-support-in-Nix) ) ## Current State From 4fa538f72a65e27b168bcc377aadc4addbf47997 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 9 Nov 2023 11:15:58 +0100 Subject: [PATCH 104/110] Call implementation comments --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 89800ff48..46f4bb9a0 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -23,7 +23,7 @@ This RFC includes two concerns that define a doc-comment: For this RFC, we adopt the following definitions: - **doc-comment**: A structured comment documenting the code's interface. -- **comment**: A free-form comment for to help with understanding the implementation and design concerns that aren't obvious. +- **implementation comment**: A free-form comment for to help with understanding the implementation and design concerns that aren't obvious. The doc-comment properties are grouped into these subcategories: From 005e34fe3c8c5b7da223fdeb26582377ddb8a6e6 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 9 Nov 2023 11:36:54 +0100 Subject: [PATCH 105/110] Update: Mental model about arguments --- rfcs/0145-doc-strings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 46f4bb9a0..d51a526b6 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -445,7 +445,7 @@ The following shows one of the many possible ways to migrate the current `nixpkg > We managed to partially automate this effort with a [codemod](https://github.com/nix-community/docnix/tree/3c0531cb5b4c9f3e9069b73d19e6c4be8508d905/codemod) -> Refactoring Note: the current `nixdoc` feature 'Function arguments' should be changed or removed. Doc-comments that relate to a lambda use this placement already. +> Note: The current `nixdoc` feature 'Function arguments' uses a different mental model than this RFC. Arguments must now be explicitly documented inside of the lambda documentation. `lib/attrsets.nix (old format)` ````nix From 6c74780b7257fd419ccfee1e4bfe7cc1be03fcec Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 9 Nov 2023 12:03:55 +0100 Subject: [PATCH 106/110] Move single line comments to future work --- rfcs/0145-doc-strings.md | 30 ++++-------------------------- 1 file changed, 4 insertions(+), 26 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index d51a526b6..02e3d3af7 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -339,32 +339,6 @@ Each subsection here contains a decision along with arguments and counter-argume -## Single-line doc-comments (not specified by this rfc) - -**Observing**: Nix offers two variants of comments; single- (`#`) and multi-line comments (`/* */`). There may be use cases where it is desirable to have a form of single-line comments subtyped for doc-comment purposes. - -**Considering**: Single-line comment for documentation. (Starting a doc-comment with `##` or similar) - -**Decision**: Single-line comments (starting with `##`) **remain unspecified** by this rfc. - -
-Arguments - -- (-) It Would be consistent with providing variants for both nix comments. -- (+) Doc-comments have only one variant to reduce complexity. -- (+) documentation will likely everytime take up more than one line. -- (+) If documentation grows bigger than one line, refactoring into a multiline-doc-comment must occur. -- (-) Offer the choice. -- (o) Single lines could also be concatenated to form multi-line documentation. - - (-) Convinience - - (+) Technical more complex -- (-) Takes up less vertical space -- (+) Visually confusing when every line starts with a `#` character. - - (+) Potential visual conflicts with the content that is markdown (e.g. with headings). -- (-) Indentation of the content is clear. - -
- # Drawbacks [drawbacks]: #drawbacks @@ -430,6 +404,10 @@ This is why we decided to follow this convention. Writing plain text is still po # Future work [Future]: #future-work +## Single-line doc-comments + +Single line doc-comment remained unspecified by this rfc. It might be an option to specify how they behave in the future. + ## Migrate existing nixpkgs comments Reformatting existing doc-comments in nixpkgs. From a9dad7b5c9bf76232694dffdc07de39f006043d1 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 9 Nov 2023 12:57:12 +0100 Subject: [PATCH 107/110] Remove partial lambda example --- rfcs/0145-doc-strings.md | 17 ----------------- 1 file changed, 17 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 02e3d3af7..ff693bae2 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -148,8 +148,6 @@ The following rules apply in descending order of precedence: - The documentation present before the `attribute path` describes the body of the attribute. ([Examples](#Attributes)) - In case placement is ambiguous, the one closer to the body has higher precedence. ([Examples](#ambiguous-placement)) - -- All partial applications of a curried lambda can share the same placement with the outermost lambda. ([Examples](#partial-lambda-functions)) ### Examples @@ -232,21 +230,6 @@ in # Documentation can still be retrieved. ``` -#### Partial lambda functions - -```nix -/**Generic doc for all partial lambda functions*/ -↓1 ↓2 ↓3 -x: ({y, ...}: z: x + y * z) - -# All partial functions can share the same placement with the outermost lambda. -# 1 -> Lambda Value `x: ({y, ...}: z: x + y * z)` -# 2 -> Lambda Value `({y, ...}: z: x + y * z)` -# 3 -> Lambda Value `z: x + y * z)` -``` - -> Allows for fallback documentation in case documentation on the partially applied function cannot be written or retrieved. - #### Lambda formals ```nix From 2987912e0f33b73b37250817953bf62fd5594a25 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Thu, 16 Nov 2023 18:16:53 +0100 Subject: [PATCH 108/110] fix typos --- rfcs/0145-doc-strings.md | 90 ++++++++++++++++++++-------------------- 1 file changed, 45 insertions(+), 45 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index ff693bae2..37b06ff63 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -23,7 +23,7 @@ This RFC includes two concerns that define a doc-comment: For this RFC, we adopt the following definitions: - **doc-comment**: A structured comment documenting the code's interface. -- **implementation comment**: A free-form comment for to help with understanding the implementation and design concerns that aren't obvious. +- **implementation comment**: A free-form comment to help with understanding the implementation and design concerns that aren't obvious. The doc-comment properties are grouped into these subcategories: @@ -48,9 +48,9 @@ The following are the envisioned goals. - Create distinct outer and inner formats for Nix doc-comments to enable accurate automated parsing and extraction. - Ensure clear differentiation from internal comments, making them accessible to tooling solutions such as documentation rendering and building. -- Convert `nixpkgs` comments intended for documentation into this format. +- Convert `Nixpkgs` comments intended for documentation into this format. -- In addition, the developer experience and adherence to established conventions should be taken into account. Equally important is ensuring that doc comments remain effortless to compose and comprehend, though it is essential to acknowledge that this aspect may vary subjectively based on personal preferences. +- In addition, the developer's experience and adherence to established conventions should be taken into account. Equally important is ensuring that doc comments remain effortless to compose and comprehend, though it is essential to acknowledge that this aspect may vary subjectively based on personal preferences. ## Non-goals @@ -58,16 +58,16 @@ The following are the envisioned goals. - How to style or format the commonmark inside of a doc-comment. -- Implementation details are not specified. The RFC shepherd group has some (feature incomplete) POC's sufficient for a generic specification. (See [Native support in Nix](#Native-support-in-Nix) ) +- Implementation details are not specified. The RFC shepherd group has some (feature incomplete) POCs sufficient for a generic specification. (See [Native support in Nix](#Native-support-in-Nix) ) ## Current State -A third-party tool called [nixdoc](https://github.com/nix-community/nixdoc) has emerged, which codifies its own rules as to the internal and external formats of a Nix doc-comment. This tool has seen some adoption, notably for the `nixpkgs.lib` functions. +A third-party tool called [nixdoc](https://github.com/nix-community/nixdoc) has emerged, which codifies its own rules as to the internal and external formats of a Nix doc-comment. This tool has seen some adoption, notably for the Nixpkgs `lib` functions. Here is an example of the format understood by *nixdoc*: ```nix - # nixpkgs/lib/trivial.nix + # Nixpkgs/lib/trivial.nix /* The constant function @@ -90,43 +90,43 @@ Here is an example of the format understood by *nixdoc*: ### Multiplicity of formats -Within nixpkgs alone, several conventions for doc-comments have emerged, see [1], [2] and [3]. +Within Nixpkgs alone, several conventions for doc-comments have emerged; see [1], [2] and [3]. -Notably, most doc-comments utilize some fraction of the of CommonMark syntax, even if they are not meant to be rendered. +Notably, most doc-comments utilize some fraction of the CommonMark syntax, even if they are not meant to be rendered. -[1]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/trivial-builders/default.nix -[2]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix -[3]: https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-disk-image.nix +[1]: https://github.com/NixOS/Nixpkgs/blob/master/pkgs/build-support/trivial-builders/default.nix +[2]: https://github.com/NixOS/Nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix +[3]: https://github.com/NixOS/Nixpkgs/blob/master/nixos/lib/make-disk-image.nix -In general, the format for writing documentation strings is **not formally specified**. +Generally, the format for writing documentation strings is **not formally specified**. -Among the formats encountered in the wild, the one used in `nixpkgs/lib` is the only one intended to be rendered as part of an API documentation, via the nixdoc third-party tool, whose syntax has not been standardized. +Among the formats encountered in the wild, the one used in `Nixpkgs/lib` is the only one intended to be rendered as part of an API documentation via the nixdoc third-party tool, whose syntax has not been standardized. ### Impossible to differentiate from internal comments -The lack of a formal definition of a doc-comment also means there is no reliable way to distinguish them from internal comments, which makes it impossible to access from doc-comments from tooling. Futhermore it makes it very hard to generate accurate and complete reference documentation. +The lack of a formal definition of a doc-comment also means there is no reliable way to distinguish them from internal comments, which makes it impossible to access doc-comments from tooling. Furthermore, it makes it very hard to generate accurate and complete reference documentation. ### References to the problems above This curated link collection highlights the Nix ecosystem's inconsistencies, a primary focus of this RFC. -#### nixpkgs - comment examples +#### Nixpkgs - comment examples -- [lib/attrsets](https://github.com/NixOS/nixpkgs/blob/5323fbf70331f8a7c47f1b4f49841cf74507f77f/lib/attrsets.nix) -- [trivial-builders](https://github.com/NixOS/nixpkgs/blob/5323fbf70331f8a7c47f1b4f49841cf74507f77f/pkgs/build-support/trivial-builders/default.nix) -- [stdenv/mkDerivation](https://github.com/NixOS/nixpkgs/blob/5323fbf70331f8a7c47f1b4f49841cf74507f77f/pkgs/stdenv/generic/make-derivation.nix) -- [nixos/lib/make-disk-image](https://github.com/NixOS/nixpkgs/blob/5323fbf70331f8a7c47f1b4f49841cf74507f77f/nixos/lib/make-disk-image.nix) +- [lib/attrsets](https://github.com/NixOS/Nixpkgs/blob/5323fbf70331f8a7c47f1b4f49841cf74507f77f/lib/attrsets.nix) +- [trivial-builders](https://github.com/NixOS/Nixpkgs/blob/5323fbf70331f8a7c47f1b4f49841cf74507f77f/pkgs/build-support/trivial-builders/default.nix) +- [stdenv/mkDerivation](https://github.com/NixOS/Nixpkgs/blob/5323fbf70331f8a7c47f1b4f49841cf74507f77f/pkgs/stdenv/generic/make-derivation.nix) +- [nixos/lib/make-disk-image](https://github.com/NixOS/Nixpkgs/blob/5323fbf70331f8a7c47f1b4f49841cf74507f77f/nixos/lib/make-disk-image.nix) # Design [design]: #detailed-design -In the following we give a comprehensive overview to our decision that we've made. Detailed arguments for and against every decision can be found in the [Decisions](#Decisions) section +In the following, we give a comprehensive overview of the decisions that we've made. Detailed arguments for and against every decision can be found in the [Decisions](#Decisions) section ## CommonMark **CommonMark according to [RFC 72](https://github.com/NixOS/rfcs/blob/master/rfcs/0072-commonmark-docs.md) is the content of all doc-comments.** -Adopting CommonMark as the content for all doc-comments brings the benefits of widely accepted and understood documentation format in tech projects, while maintaining profitability and consistency within the Nix ecosystem by aligning with existing [NixOS/RFC-72](https://github.com/NixOS/rfcs/blob/master/rfcs/0072-commonmark-docs.md). +Adopting CommonMark as the content for all doc-comments brings the benefits of widely accepted and understood documentation format in tech projects while maintaining profitability and consistency within the Nix ecosystem by aligning with existing [NixOS/RFC-72](https://github.com/NixOS/rfcs/blob/master/rfcs/0072-commonmark-docs.md). ## `/** */` is the doc-comment format @@ -317,8 +317,8 @@ Each subsection here contains a decision along with arguments and counter-argume - (+) Allows copy-paste from and to markdown files. We allow easy refactoring if documentation arises and needs to be split into separate files. - (-) Strictly binding doc comments content to commonMark might restrict users. - (+) Users/Tools can still use regular comments or develop alternative solutions. -- (-) CommonMark does not specify the current rich features from nixdoc, such as predefined sections and structures that could be used as a source for automated toolings. Such as types for type-checking or examples to run automated-tests - - (+) Future tools can still build their conventions on top of this RFC. They might not directly specify them in an RFC but be a tool developers choose for a specific codebase. However, we have yet to get that tools. So it is good when rich features remain unspecified. +- (-) CommonMark does not specify the current rich features from nixdoc, such as predefined sections and structures that could be used as a source for automated toolings. Such as types for type-checking or examples to run automated tests + - (+) Future tools can still build their conventions on top of this RFC. They might not directly specify them in an RFC but be a tool developers choose for a specific codebase. However, we have yet to get those tools. So, it is good when rich features remain unspecified. @@ -329,15 +329,15 @@ Each subsection here contains a decision along with arguments and counter-argume This could be (partially) automated. (see our [codemod](https://github.com/nix-community/docnix/tree/3c0531cb5b4c9f3e9069b73d19e6c4be8508d905/codemod) ) -Also, the migration could be performed piecemal, starting perhaps with `nixpkgs.lib`. +Also, the migration could be performed piecemeal, starting perhaps with Nixpkgs `lib`. See future work. -## Breaking the nixpkgs manual +## Breaking the Nixpkgs manual -This is a breaking change, to the current nixpkgs manual tooling Nixpkgs library function documentation. +This is a breaking change to the current Nixpkgs manual tooling Nixpkgs library function documentation. -See future work. +Please take a look at future work. # Alternatives [alternatives]: #alternatives @@ -346,7 +346,7 @@ See future work. | Property / Approach | `##` | `/** */` | `Javadoc` | `/*\|` or `/*^` | All comments are doc-comments | |---|---|---|---|---|---| -| Inspired by | Rust | Current nixpkgs.lib | C++/Java/Javascript | Haskell Haddock | Current nixpkgs.lib | +| Inspired by | Rust | Current Nixpkgs `lib` | C++/Java/JavaScript | Haskell Haddock | Current Nixpkgs.lib | | Changes the existing code by | Much | Less | Even More | Less | ? | | Needs Termination | No | Yes | Yes | Yes | ? | | Indentation | Clear | like Nix's multiline strings, thus **Intuitive** | Clear | ? | ? | @@ -362,14 +362,14 @@ See future work. **Observing**: From a refactoring perspective, it might also be interesting to see how many conflicts the different formats would cause. -nixpkgs comments: +Nixpkgs comments: - `##` ~4k usages - `#` ~20k usages - `/*` ~6k usages - `/**` 160 usages -Choosing `/**` or subsets would cause minor conflicts within current nixpkgs. While this is NOT the main reason for the final decision, it MUST be considered. +Choosing `/**` or subsets would cause minor conflicts within current Nixpkgs. While this is NOT the main reason for the final decision, it MUST be considered. ## Just free text as a content format @@ -381,19 +381,19 @@ This is why we decided to follow this convention. Writing plain text is still po # Unresolved questions [unresolved]: #unresolved-questions -- Migration path for nixpkgs comments. -- How to document the `arguments`. Should there be some markdown equivalent to `@param` i.e. `# Params`? This RFC intentionally leaves this question unanswered, allowing for further discussion and decision-making in the future. +- Migration path for Nixpkgs comments. +- How to document the `arguments`. Should there be some markdown equivalent to `@param`, i.e. `# Params`? This RFC intentionally leaves this question unanswered, allowing for further discussion and decision-making in the future. # Future work [Future]: #future-work ## Single-line doc-comments -Single line doc-comment remained unspecified by this rfc. It might be an option to specify how they behave in the future. +Single-line doc-comment remained unspecified by this RFC. It might be an option to specify how they behave in the future. -## Migrate existing nixpkgs comments +## Migrate existing Nixpkgs comments -Reformatting existing doc-comments in nixpkgs. +Reformatting existing doc-comments in Nixpkgs. Action points: @@ -402,7 +402,7 @@ Action points: ### Migration Example -The following shows one of the many possible ways to migrate the current `nixpkgs.lib` comments. +The following shows one of the many possible ways to migrate the current Nixpkgs `lib` comments. > We managed to partially automate this effort with a [codemod](https://github.com/nix-community/docnix/tree/3c0531cb5b4c9f3e9069b73d19e6c4be8508d905/codemod) @@ -421,7 +421,7 @@ The following shows one of the many possible ways to migrate the current `nixpkg filterAttrs :: (String -> Any -> Bool) -> AttrSet -> AttrSet */ filterAttrs = - # Predicate taking an attribute name and an attribute value, which returns `true` to include the attribute, or `false` to exclude the attribute. + # Predicate taking an attribute name and an attribute value, which returns `true` to include the attribute or `false` to exclude the attribute. pred: # The attribute set to filter set: @@ -464,24 +464,24 @@ filterAttrs = All current rendering tooling solutions should support displaying the specified doc-comment format. -Currently at least: +Currently, at least: -### nixpkgs Manual renderer +### Nixpkgs Manual renderer -The current nixpkgs manual needs to be adopted to this change. +The current Nixpkgs manual needs to be adapted to this change. -We expect changes in the following to be neccessary: +We expect changes in the following to be necessary: -- [nixos_render_docs](https://github.com/NixOS/nixpkgs/tree/e4082efedb483eb0478c3f014fa851449bca43f9/pkgs/tools/nix/nixos-render-docs/src) +- [nixos_render_docs](https://github.com/NixOS/Nixpkgs/tree/e4082efedb483eb0478c3f014fa851449bca43f9/pkgs/tools/nix/nixos-render-docs/src) - [nixdoc](https://github.com/nix-community/nixdoc) ## Native support in Nix - `NixOS/nix` should implement native support for doc-comments. -Primarily because the *Nix* language requires access to the actual evaluator for building correct relations between doc-comment and and the node they document. +Primarily because the *Nix* language requires access to the actual evaluator for building correct relations between doc-comment and the node they document. -> Note: We considered implementation details, but specifying those is out of scope for this rfc. +> Note: We considered implementation details, but specifying those is out of the scope of this RFC. Related POCs: @@ -516,4 +516,4 @@ Related POCs: - https://github.com/NixOS/nix/pull/5527 - https://github.com/NixOS/nix/pull/1652 -[#1652](https://github.com/NixOS/nix/pull/1652) gets closest what this RFC imagines, implementation is almost done. It certainly has some limitations, but was not merged due to uncertainty and complexity. +[#1652](https://github.com/NixOS/nix/pull/1652) gets closest to what this RFC imagines, implementation is almost done. It has some limitations but was not merged due to uncertainty and complexity. From e4a6dd72767fcf037f68e3e33fa5f03a5827c9ee Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Sun, 19 Nov 2023 11:20:45 +0100 Subject: [PATCH 109/110] nix implementation is out of scope. thanks @amjoseph-nixpkgs. Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com> --- rfcs/0145-doc-strings.md | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 37b06ff63..24de0788c 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -475,21 +475,6 @@ We expect changes in the following to be necessary: - [nixos_render_docs](https://github.com/NixOS/Nixpkgs/tree/e4082efedb483eb0478c3f014fa851449bca43f9/pkgs/tools/nix/nixos-render-docs/src) - [nixdoc](https://github.com/nix-community/nixdoc) -## Native support in Nix - -- `NixOS/nix` should implement native support for doc-comments. - -Primarily because the *Nix* language requires access to the actual evaluator for building correct relations between doc-comment and the node they document. - -> Note: We considered implementation details, but specifying those is out of the scope of this RFC. - -Related POCs: - -- https://github.com/NixOS/nix/pull/9054 -- https://github.com/NixOS/nix/pull/1652 -- https://github.com/NixOS/nix/pull/5527 -- https://github.com/NixOS/nix/commit/8e252320f52426c734883709eb398d2161c3fe82 - ## References ### Other Conventions From 90d219b91f7a336aeca9c8a96a3aa470e367d0b6 Mon Sep 17 00:00:00 2001 From: Johannes Kirschbauer Date: Mon, 20 Nov 2023 09:54:13 +0100 Subject: [PATCH 110/110] Add occurence details about /**/ and /** */ --- rfcs/0145-doc-strings.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/rfcs/0145-doc-strings.md b/rfcs/0145-doc-strings.md index 24de0788c..1b7e0a74e 100644 --- a/rfcs/0145-doc-strings.md +++ b/rfcs/0145-doc-strings.md @@ -367,9 +367,11 @@ Nixpkgs comments: - `##` ~4k usages - `#` ~20k usages - `/*` ~6k usages -- `/**` 160 usages +- `/**` 160 usages + - `/**{content}*/` 10 usages. Need to be migrated, if using different convention. + - `/**/` 35 usages. They still are non-doc-comments -Choosing `/**` or subsets would cause minor conflicts within current Nixpkgs. While this is NOT the main reason for the final decision, it MUST be considered. +Choosing `/**` or subsets would cause minor conflicts within current Nixpkgs. While this is NOT the main reason for the final decision, it must be considered. ## Just free text as a content format