EDoc: EEP-48 doc chunk generation #2803

erszcz · 2020-10-14T15:09:14Z

This PR enables EDoc to output EEP-48 style doc chunks. The work leading to this PR was so far discussed and documented at erlef/documentation-wg#4, whereas developed at https://github.com/erszcz/edoc.

I tried to address all points raised in the discussions so far:

Chunks are written to disk as standalone files, not embedded into .beam files.
It's possible to generate chunks from the command line by running a script.
The chunks are as close to erl_docgen chunks as possible (with the caveat of type links described below).
All @spec and @type tags are rewritten to -spec and -type attributes, respectively.

We discussed the type link caveat with @garazdawi - erl_docgen chunk links to types do not use type arities. This means links to t/0 and t/1 have the same form. EDoc chunks do not follow this principle, since it seemed more robust to allow linking to arbitrary types. Until this is fixed for erl_docgen chunks, measures have to be taken in 3rd party tools (e.g. ExDoc) to watch for this difference.

Some features stemming from EDoc design and implementation, but not necessarily intuitive, are not addressed in this PR:

Function doc comments can precede or follow -spec attributes, but type doc comments must follow -type attributes.
@since and @deprecated tags are supported on functions, but are not supported on types or callbacks.

This PR also adds test/eep48_SUITE.erl which tests the majority of the newly added functionality, as well as some of the above points which are currently not supported (these tests are skipped by default).

josevalim · 2020-10-14T17:22:51Z

This is amazing work @erszcz! ❤️

One of the things to discuss, probably for after this PR is merged, is to deprecate the tags @spec and @type in EDoc. Possibly stop parsing them altogether and emit a warning, since they won't be used by chunks and, afaik, the offer not benefit (only downsides?) over the existing -type and -spec.

KennethL · 2020-10-15T06:03:36Z

I was just about to ask if we could get this in place for the OTP 24 release (and release candidates beginning in February) so this is fantastic good news. Thanks @erszcz. /Regards Kenneth

…

garazdawi

I've done a cursory review and I think all in it it looks very good. Most of the comments are about backward-compatibility and documentation.

It would be nice if the changes of @spec to -spec would be a separate commit from everything else so that it is easier to see that it does not change any functionality.

Does @richcarl have any comments about this PR?

I also looked through some rendered functions and found some small irregularities that I'm not sure if you have seen or not:

erl_parse:tree/2

In the edoc html generate this function looks like this:

tree(Type::atom(),Data::term()) -> tree()

however, in the shell docs it looks like this:

-spec tree(atom(),term()) -> tree().

This because edoc for html reads the variable names and inserts them
into the spec. Information is lost when this is not done as the reader
does not know which argument is Type and which is Data in the text.

-spec tree(atom(), term()) -> tree().

For special purposes only. Creates an abstract syntax tree node
with type tag *Type* and associated data *Data*.

Problem in shell_docs

Seems like shell_docs does not render <pre> correctly, or rather
it does not insert a newline before itself so that things like merl
look like this:

  Quick start

  To enable the full power of Merl, your module needs to include the
  Merl header file:
         -include_lib("syntax_tools/include/merl.hrl").

  Then, you can use the ?Q(Text) macros in your code to create
  ASTs or match on existing ASTs. For example:
         Tuple = ?Q("{foo, 42}"),
         ?Q("{foo, _@Number}") = Tuple,
         Call = ?Q("foo:bar(_@Number)")

  Calling merl:print(Call) will then print the following code:
         foo:bar(42)

  The ?Q macros turn the quoted code fragments into ASTs, and

I'll take a look at fixing that and try not to break any of the erl_docgen renderings.

lib/edoc/bin/edoc.escript

lib/edoc/doc/overview.edoc

lib/edoc/bin/edoc.escript

lib/edoc/doc/overview.edoc

lib/edoc/src/edoc_layout.erl

lib/edoc/src/edoc_run.erl

lib/edoc/test/eep48_SUITE.erl

lib/stdlib/src/shell_docs.erl

lib/edoc/src/edoc.erl

erszcz · 2020-12-08T09:39:25Z

Thanks for the review, @garazdawi!

I think it makes the most sense to start with the spec/type changes. I'm sending a separate PR shortly, to make it easy to review. Then, I'll rebase this as appropriate.

erszcz · 2020-12-11T14:21:31Z

With regard to erl_syntax:tree/1,2, I've added code which inserts argument names into the specs stored in the chunk. This means the spec in the chunk will not reflect the spec in the source file 1-to-1 anymore, but for the example from erl_syntax.erl:

-spec tree(atom()) -> tree().

tree(Type) ->
    tree(Type, []).

-spec tree(atom(), term()) -> tree().

tree(Type, Data) ->
    ...

We get properly named types in erl:

1> h(erl_syntax, tree).

  -spec tree(Type :: atom()) -> tree().

  Equivalent to tree(Type, []).

  -spec tree(Type :: atom(), Data :: term()) -> tree().

  For special purposes only. Creates an abstract syntax tree node with type tag Type and
  associated data Data.

It does not yet support 'bounded_fun' syntax tree nodes (-spec f(A) -> A when A :: type()) - for now, these are just passed as is.

erszcz · 2020-12-18T13:32:44Z

These should probably be grouped into one:
See also data/1, is_tree/1, type/1

Done in the newest batch of changes.

Moreover, arg names are now properly added to bounded_fun kind of specs.

garazdawi · 2021-01-07T10:35:22Z

Would you mind rebasing on top of latest master to get the #2914 changes removed from this PR?

erszcz · 2021-01-07T16:42:33Z

@garazdawi Sure, I should be able to rebase this tomorrow. I also have some WIP on the documentation for the escript - I'll attach on top of the rebase.

garazdawi · 2021-01-08T08:06:02Z

Great, let me know when you are ready for me to review the PR again.

erszcz · 2021-01-08T13:29:15Z

I've rebased this on top of the current master. I think I've addressed all points of the first review - please let me know if there's something missing. The current status is as follows:

edoc.escript is renamed to just edoc - I think it's simpler and better, if it's intended for public use. However, in the HTML table of contents it is listed as edoc_cmd, since otherwise it would clash with edoc.erl docs. I think it's fine this way, but let me know if you think otherwise.
Specs stored in chunks don't reflect 1-to-1 the specs in source files - they're extended to contain arg names apart from types. I think that you, @garazdawi, intended to make the specs in source files and in the chunks to be 100% identical. Maybe EDoc should just warn "Hey, your spec looks like this: ..., but it'd look better like that: .... Would you mind fixing your source?" That would only require the user to copy-n-paste the rewritten example, but I'm afraid even that might be too much. This is borderline with EDoc being a linter. Opinions are welcome.
@josevalim's good point about warnings for @type and @spec. This is slightly connected with the above point. Should this warning be part of this PR?
If @spec and @type are considered deprecated, then what about deprecating @deprecated (pun intended ;) )? There's xref which supports a -deprecated attribute, so this case is analogous to dialyzer and @spec/@type. I think it's worth considering, but not necessarily as part of this PR. This would also solve the problem of EEP-48 defining the deprecated field as just a flat binary(), while current EDoc allowing it to contain markup.

TODOs:

When testing this rebased code on Recon, I found out that user_type nodes (that is all non-builtin types) are not handled in the spec rewriting code - I'll work on this next.

garazdawi · 2021-01-11T10:25:26Z

Hello,

I did lib/edoc/bin/edoc -chunks -app syntax_tools and got a bunch of crashes looking like this:

edoc: error in layout 'edoc_layout_chunks': {'EXIT',
                                       {function_clause,
                                        [{edoc_layout_chunks,annotate_spec,
                                          [['Name'],undefined],
                                          [{file,"edoc_layout_chunks.erl"},
                                           {line,250}]},

Is that because they have both a -spec and @spec?

garazdawi · 2021-01-11T10:31:42Z

Specs stored in chunks don't reflect 1-to-1 the specs in source files - they're extended to contain arg names apart from types. I think that you, @garazdawi, intended to make the specs in source files and in the chunks to be 100% identical.

I think it is fine that they do not match 100%. This is an oddity in the way that edoc behaves, but I don't think it is worth forcing users to rewrite their specs.

garazdawi · 2021-01-11T10:37:53Z

warnings for @type and @spec. This is slightly connected with the above point. Should this warning be part of this PR?

I think it is better to do this in another PR.

garazdawi

I did a quick look to see if it would be easy to add support in the make system to use edoc instead of erl_docgen to build the doc chunks for Erlang/OTP and it turns out that it is not. So I think we will leave that for later.

My only remaining question from the original review is whether we should document the edoc_doclet_chunks and edoc_layout_chunks modules? The way it is implemented now it is only possible to generate chunks via the script without relying on undocumented API's.

lib/edoc/src/edoc_layout_chunks.erl

lib/edoc/doc/overview.edoc

erszcz · 2021-01-20T12:16:24Z

Status update:

When testing this rebased code on Recon, I found out that user_type nodes (that is all non-builtin types) are not handled in the spec rewriting code - I'll work on this next.

Done.

got a bunch of crashes ... because they have both a -spec and @SPEC?

Indeed, that turns out to be the reason. It seems that in such a case EDoc doesn't pass the -spec attr form down to the layout :| I might be able to find out why and fix it in EDoc core, but for now a warning will be printed that just @spec or redundant -spec and @spec are not supported when generating chunks. At least we have a clear answer to the "should we print a warning or not" question :)

In order to gather all info about a type that EEP-48 requires, we refer to EDoc #tag{} records. These records do not contain the type name at the top level, so this lookup was done just by line number. When a header file is included into another file, it might happen that the position information of two distinct type definitions is the same - they are still different types, but the lookup by line will return the wrong one. This lead to the following error (on kernel/src/disk_log.erl): name: dlog_size arity: 0 type attr: {attribute,82,type,{file_error,{type,82,term,[]},[]}} edoc: error in layout 'edoc_layout_chunks': {'EXIT', {{badmatch,{file_error,[]}}, [{edoc_layout_chunks,meta_type_sig,4, [{file,"edoc_layout_chunks.erl"}, {line,170}]}, This change introduces a more sophisticated and robust lookup, which fixes the problem of types defined in include files.

- Fix unused variables and lookup functions - Remove edoc-orig-tag - Remove name attribute from <a> elements - Be less verbose if not debugging

Co-authored-by: Lukas Larsson <[email protected]>

This change also prevents the 'multiple @SPEC tag' error.

garazdawi · 2021-02-08T11:06:25Z

github actions doc build has started to fail with the latest changes, do you know why?

erszcz · 2021-02-08T12:17:19Z

ce77e68 and c5581d2 introduce the priority of attrs over tags. However, as this changes EDoc core, it's not limited to chunk generation, but all paths, so static HTML or erl_docgen spec extraction, too.

Specs were missing in OTP docs after these changes -
93c34bb fixes that.

I'm now troubleshooting some types which used to be exported, but currently are not.

erszcz · 2021-02-08T12:52:08Z

I’ve reverted the PR to the last commit passing all checks. I’ll carry on in a separate branch.

Apparently, these are not handled properly by EDoc, therefore types mentioned in them aren't properly exported to docs.

erszcz · 2021-02-08T17:59:58Z

The latest commits invert the priorities of tags and attributes. -spec and -type now take precedence over @spec and @type. Warnings will be emitted for the tags.

This priority inversion means that if a module has both a @type and a -type of the same name, but only the former carries a doc comment, then this comment will not appear anymore in either the HTML or chunk output. See @type array in stdlib/src/array.erl for an example. This is a backward incompatible change, but should not affect OTP docs.

In case of specs, the problem of consecutive -spec attributes is solved - EDoc can now accept specs in any order and not necessarily directly adjacent to a function they describe. This solves some errors in OTP files, e.g. stdlib/src/otp_internal.erl.

I finally think there's nothing more to add to this PR, unless some bugs pop up. @garazdawi @richcarl what do you think?

garazdawi · 2021-02-10T08:02:21Z

I finally think there's nothing more to add to this PR, unless some bugs pop up.

I agree.

garazdawi · 2021-02-12T08:37:35Z

Merged! Thanks for all your work!

erszcz · 2021-02-15T09:52:15Z

Woohoo! And thank you for the reviews!

erszcz mentioned this pull request Oct 14, 2020

Generate ExDoc documentation for Erlang projects erlef/documentation-wg#4

Open

16 tasks

rickard-green added the team:VM Assigned to OTP team VM label Oct 19, 2020

erszcz mentioned this pull request Nov 13, 2020

Generate docs for Erlang projects elixir-lang/ex_doc#1291

Closed

garazdawi self-assigned this Nov 23, 2020

garazdawi added the feature label Nov 23, 2020

garazdawi reviewed Nov 23, 2020

View reviewed changes

garazdawi mentioned this pull request Nov 23, 2020

Shell docs fixes #2881

Merged

erszcz mentioned this pull request Dec 8, 2020

EDoc spec cleanup #2914

Merged

erszcz force-pushed the edoc-chunk-support branch from fc009ab to f1c92fa Compare December 9, 2020 09:16

erszcz changed the title ~~EEP-48 doc chunk support for EDoc~~ EDoc: EEP-48 doc chunk generation Dec 9, 2020

erszcz force-pushed the edoc-chunk-support branch from 700bf56 to eafc139 Compare December 10, 2020 15:26

garazdawi added the waiting waiting for changes/input from author label Jan 8, 2021

erszcz force-pushed the edoc-chunk-support branch 2 times, most recently from e10d3cb to db33aff Compare January 8, 2021 12:34

garazdawi reviewed Jan 11, 2021

View reviewed changes

lib/edoc/src/edoc_layout_chunks.erl Outdated Show resolved Hide resolved

lib/edoc/src/edoc_layout_chunks.erl Outdated Show resolved Hide resolved

lib/edoc/src/edoc_layout_chunks.erl Outdated Show resolved Hide resolved

lib/edoc/doc/overview.edoc Outdated Show resolved Hide resolved

erszcz force-pushed the edoc-chunk-support branch from d8b83b9 to bc0d6d4 Compare January 20, 2021 12:15

garazdawi mentioned this pull request Jan 20, 2021

Allow h4,h5 and h6 headers in application/erlang+html EEP-48 documentation #2991

Merged

erszcz and others added 11 commits February 5, 2021 21:06

Add edoc/.gitignore

4001143

Allow all header ranks in the chunks

2092cc2

Improve spec formatter completeness, diagnostics and robustness

f0f14ef

Fix various small things

4a58810

- Fix unused variables and lookup functions - Remove edoc-orig-tag - Remove name attribute from <a> elements - Be less verbose if not debugging

Add a script to batch test libraries

921436e

Allow only HTML tags allowed by shell_docs

461ef2b

Document EEP-48 support, making doc chunks, new doclet/layout

1343415

Depend on stdlib with shell_docs

722328c

Co-authored-by: Lukas Larsson <[email protected]>

Change the description

3728fac

Adjust function signature formatting to better comply with EEP-48

52dc947

erszcz force-pushed the edoc-chunk-support branch from ea0046d to 52dc947 Compare February 5, 2021 22:08

erszcz added 3 commits February 7, 2021 14:22

Warn about deprecated @SPEC tags and prioritise -spec attributes

ce77e68

This change also prevents the 'multiple @SPEC tag' error.

Warn about deprecated @type tags and prioritise -type attributes

c5581d2

Handle function spec clauses in docgen_otp_specs

93c34bb

erszcz force-pushed the edoc-chunk-support branch from 93c34bb to 52dc947 Compare February 8, 2021 12:51

erszcz added 3 commits February 8, 2021 14:34

Add a fixture of a spec prefix with a module name

dd975bb

Apparently, these are not handled properly by EDoc, therefore types mentioned in them aren't properly exported to docs.

Handle specs prefixed by a module name

60f7634

Do not drop module name from MFA-style specs

a347941

garazdawi merged commit a347941 into erlang:master Feb 12, 2021

erszcz mentioned this pull request Mar 31, 2021

EDoc & co. - known issues and next steps erlef/build-and-packaging-wg#25

Closed

garazdawi mentioned this pull request Oct 4, 2021

Eep 48 markdown erlang-ls/erlang_ls#1100

Merged

9 tasks

erszcz mentioned this pull request Sep 7, 2023

Update EEP statuses erlang/eep#52

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EDoc: EEP-48 doc chunk generation #2803

EDoc: EEP-48 doc chunk generation #2803

erszcz commented Oct 14, 2020 •

edited

Loading

josevalim commented Oct 14, 2020

KennethL commented Oct 15, 2020 via email

garazdawi left a comment •

edited

Loading

erszcz commented Dec 8, 2020

erszcz commented Dec 11, 2020

erszcz commented Dec 18, 2020

garazdawi commented Jan 7, 2021

erszcz commented Jan 7, 2021

garazdawi commented Jan 8, 2021

erszcz commented Jan 8, 2021 •

edited

Loading

garazdawi commented Jan 11, 2021

garazdawi commented Jan 11, 2021

garazdawi commented Jan 11, 2021 •

edited

Loading

garazdawi left a comment

erszcz commented Jan 20, 2021

garazdawi commented Feb 8, 2021

erszcz commented Feb 8, 2021

erszcz commented Feb 8, 2021

erszcz commented Feb 8, 2021

garazdawi commented Feb 10, 2021

garazdawi commented Feb 12, 2021

erszcz commented Feb 15, 2021

EDoc: EEP-48 doc chunk generation #2803

EDoc: EEP-48 doc chunk generation #2803

Conversation

erszcz commented Oct 14, 2020 • edited Loading

josevalim commented Oct 14, 2020

KennethL commented Oct 15, 2020 via email

garazdawi left a comment • edited Loading

Choose a reason for hiding this comment

erl_parse:tree/2

See also

Problem in shell_docs

erszcz commented Dec 8, 2020

erszcz commented Dec 11, 2020

erszcz commented Dec 18, 2020

garazdawi commented Jan 7, 2021

erszcz commented Jan 7, 2021

garazdawi commented Jan 8, 2021

erszcz commented Jan 8, 2021 • edited Loading

garazdawi commented Jan 11, 2021

garazdawi commented Jan 11, 2021

garazdawi commented Jan 11, 2021 • edited Loading

garazdawi left a comment

Choose a reason for hiding this comment

erszcz commented Jan 20, 2021

garazdawi commented Feb 8, 2021

erszcz commented Feb 8, 2021

erszcz commented Feb 8, 2021

erszcz commented Feb 8, 2021

garazdawi commented Feb 10, 2021

garazdawi commented Feb 12, 2021

erszcz commented Feb 15, 2021

erszcz commented Oct 14, 2020 •

edited

Loading

garazdawi left a comment •

edited

Loading

erszcz commented Jan 8, 2021 •

edited

Loading

garazdawi commented Jan 11, 2021 •

edited

Loading