Generate role documentation #272

felixfontein · 2021-05-21T07:45:00Z

With the role argument spec feature of ansible-core 2.11, it is now possible to document roles in collections!

~~This is a first WIP to implement this. It's also a WIP because it contains #255.~~ There's a lot of special-casing since roles behave quite differently than other plugin types (for example they can have multiple entry points).

You can see the effects here:

The menu contains 'Index of all Roles': https://ansible.fontein.de/
There's a 'Index of all Roles': https://ansible.fontein.de/collections/index_role.html
In the felixfontein.acme collection docs, there's a 'Role Index': https://ansible.fontein.de/collections/felixfontein/acme/index.html#role-index (please ignore the 'Roles' section above, it is created manually with Allow collections to add RST files #255)
There are role documentations:

felixfontein · 2021-05-25T16:32:18Z

I also built documentation for sensu.sensu_go, a collection included in Ansible which has roles with argument specs: https://ansible.fontein.de/collections/sensu/sensu_go/index.html#role-index

Shrews · 2021-05-28T20:45:51Z

antsibull/data/collection-enum.py

@@ -195,6 +229,13 @@ def main(args):
    for plugin_type in C.DOCUMENTABLE_PLUGINS:
        result['plugins'][plugin_type] = load_all_plugins(plugin_type, basedir, coll_filter)

+    # Export role docs
+    RoleMixin = getattr(doc, 'RoleMixin', None)


Just FYI, neither RoleMixin nor any of its methods were ever meant to be publicly accessible. Using this is at-your-own-risk as my intention was to try to make a proper public API for some of this in the future.

Actually most code in collection-enum.py is using internal stuff from ansible-doc that can break at any moment. I would prefer to have a stable API, but unfortunately there isn't one :)

(In case it breaks, you can configure antsibull to use ansible-doc --json to extract all the docs. Unfortunately it's like 1000x slower, that's why this method is the default :) )

wbclark · 2021-06-08T16:42:50Z

Hi @felixfontein ! Thanks for the work here on structured docs for Ansible roles

We would like to use this approach for our roles in the Foreman community collections. We are very interested in your feedback on the following ideas and whether you consider it to be a good approach, feasible as already implemented, or worth implementing here:

Adding a template for README.md in the roles/$role_name directory to be populated from the structured documentation.
A way to include 'Common Role Variables' which are shared across multiple roles but defined in a single location such as parent roles/ directory for the collection. To make things conceptually simple for users, these would ideally be rendered under a separate heading.
A way to define groupings of Role Variables, assign a description to each grouping, and render the groupings separately in the docs. For example we might want separate groupings for required parameters, parameters which have values in the role defaults, parameters which have no default in the role but do have a default value in the target, and parameters which will be omitted in the target if not specified.
A way to define Example Playbooks for the Role within an example_playbooks/ directory, and render each of these along with a description in the documentation.

Most of our roles accept a primary data structure which is usually a single yaml dictionary, or sometimes a list of yaml dictionaries (so the 'groupings' I refer to above would ideally be possible at the level of sub-parameters for the yaml dictionary). My current understanding (please correct me if I am wrong) is that class InnerRoleOptionsSchema allows for handling of nested data structures like yaml lists and dictionaries. If that is correct, it would be great to see a test case / example!

If you like these ideas and consider them to be worthwhile, we would of course be happy to assist with implementation. Looking forward to your feedback.

CC: @evgeni for any additional comments

felixfontein · 2021-06-08T19:35:43Z

Hi @wbclark, thanks for your interest in this PR!

1. Adding a template for README.md in the `roles/$role_name` directory to be populated from the structured documentation.

This should be doable, but I'm not sure whether it should be part of antsibull (and definitely not part of this PR :) ).

2. A way to include 'Common Role Variables' which are shared across multiple roles but defined in a single location such as parent `roles/` directory for the collection.

This could be solved with something similar to docs fragments, but unfortunately ansible-core does not support that (yet). It would be nice if this could be added for ansible-core 2.12, but this repository is the wrong venue to discuss this (it's the core team's decision, not ours). It's probably a good idea to discuss this with core team members in #ansible-devel and/or a core meeting, and/or creating a proposal (https://github.com/ansible/proposals/) for that.

To make things conceptually simple for users, these would ideally be rendered under a separate heading.

This would require additional metadata in the argument spec. While this is probably more relevant for roles, it can also be of interest for plugins. It's probably best to start discussing this in the documentation working group meeting. This is in particular important since the ansible-doc text output should have a similar amount of information than the generated RST documentation.

3. A way to define groupings of Role Variables, assign a description to each grouping, and render the groupings separately in the docs. For example we might want separate groupings for required parameters, parameters which have values in the role defaults, parameters which have no default in the role but do have a default value in the target, and parameters which will be omitted in the target if not specified.

This is similar to the above: it needs additional metadata, which needs to be defined and standardized first (preferably in the docs meeting and/or as a proposal).

4. A way to define Example Playbooks for the Role within an `example_playbooks/` directory, and render each of these along with a description in the documentation.

This also needs to be defined and standardized first, so that both ansible-doc and antsibull-docs process data the same way. (I cannot say whether reading examples from external files would be acceptable for ansible-doc though.)

Most of our roles accept a primary data structure which is usually a single yaml dictionary, or sometimes a list of yaml dictionaries (so the 'groupings' I refer to above would ideally be possible at the level of sub-parameters for the yaml dictionary).

That's not so easy to implement with the way module, plugin and role options are presented currently (as a table). I'm planning to work on a different representation eventually (which will not use <table>), with that this is probably easier to implement. It still needs additional metadata though.

My current understanding (please correct me if I am wrong) is that class InnerRoleOptionsSchema allows for handling of nested data structures like yaml lists and dictionaries. If that is correct, it would be great to see a test case / example!

Yes, nested data is supported, see the specification: https://docs.ansible.com/ansible/devel/user_guide/playbooks_reuse_roles.html#specification-format It looks like suboptions: is the wrong key though (i.e. the documentation is wrong about this, and also this PR until 8e92ab3) - the correct key seems to be options: according to ansible-core's code and its integration tests. I haven't seen an example of its use outside the integration tests (https://github.com/ansible/ansible/tree/devel/test/integration/targets/roles_arg_spec) yet, though.

antsibull/docs_parsing/ansible_doc.py

antsibull/data/docsite/role.rst.j2

samccann

just a few questions since I can't follow the code....;-)

The output so far LGTM.

samccann · 2021-07-08T19:29:32Z

antsibull/data/docsite/role.rst.j2

+
+{% if collection == 'ansible.builtin' -%}
+.. note::
+    This module is part of ``ansible-base`` and included in all Ansible


Should we be saying it's part of 'ansible-core' at this point? Also, since I'm not really following the code, is it a module that triggers this note, or is it some future role included in ansible.builtin?

I cut this paragraph together to

This role is part of ``ansible-core`` and included in all Ansible installations.

This case will probably not happen in the short term anyway (and maybe never in the long term as well); I think ansible-core right now doesn't even have a mechanism by which it can contain roles.

antsibull/data/docsite/role.rst.j2

samccann · 2021-07-08T19:35:50Z

antsibull/data/docsite/role.rst.j2

+.. seealso::
+
+{%     for item in ep_doc['seealso'] %}
+{%       if item.module is defined and item.description %}


I don't know that any of the example roles are exercising this bit of the code yet. Is there a way to 'dummy up' some See Also sections so we can be sure this part works?

Here's now an example: https://ansible.fontein.de/collections/felixfontein/acme/acme_certificate_role.html#see-also

acozine

Thanks for implementing role documentation @felixfontein! This is a big addition, and I think could be useful. I have a few questions, see below.

antsibull/data/collection-enum.py

acozine · 2021-07-12T19:25:03Z

antsibull/data/docsite/plugins_by_collection.rst.j2

@@ -29,20 +29,35 @@ Collection version @{ collection_version }@

 {% endfor %}

+{% if 'role' in plugin_maps %}
+Role Index


I wonder if this is the correct order. For a majority of collections, most users are looking for modules. In the docs survey results, we had at least one person say that they were confused by the Plugin Index heading and thought they were on the wrong page. For a collection that includes modules, other plugins, and roles, I would probably expect that to be the order of presentation (modules first, then plugins, then roles).

I moved it to the bottom in c28c8e4.

Maybe we should rename "Plugin Index" to "Module and Plugin Index"?

That's a good idea. Or "Module and Other Plugin Index"? Modules really are just a type of plugin . . . but I don't know how important it is for people to know that . . .

That's a good idea. Or "Module and Other Plugin Index"? Modules really are just a type of plugin . . . but I don't know how important it is for people to know that . . .

+1 to changing to this.

antsibull/data/docsite/role.rst.j2

acozine · 2021-07-12T19:28:06Z

antsibull/data/docsite/role.rst.j2

+
+.. Collection note
+
+{% if collection == 'ansible.builtin' -%}


Will we ever have roles in ansible core?

No idea. I don't think this will happen short-term, and probably never long-term either. But if it does happen, the description will be less wrong than if this isn't specially handled :)

antsibull/data/docsite/role.rst.j2

acozine · 2021-07-12T19:31:34Z

antsibull/data/docsite/role.rst.j2

+^^^^^^^
+
+{%     for author_name in ep_doc['author'] %}
+- @{ author_name }@


Do we. have this metadata?

This section is only added when author is present. Some roles do provide this.

acozine · 2021-07-12T19:37:55Z

antsibull/write_docs.py

    error_tmpl = env.get_template('plugin-error.rst.j2')

    writers = []
    lib_ctx = app_context.lib_ctx.get()
    async with asyncio_pool.AioPool(size=lib_ctx.thread_max) as pool:
        for collection_name, plugins_by_type in collection_to_plugin_info.items():
            for plugin_type, plugins in plugins_by_type.items():
+                plugin_type_tmpl = plugin_tmpl
+                if plugin_type == 'role':


This is probably the simplest way to extend the existing docs, but . . . are roles actually plugins? If they are, then this is fine. If they aren't, this seems like adding confusion for Future Coders. If roles aren't plugins, then could we use a different name? Like "elements" or something - where all plugins would be elements, and roles would also be elements . . . ?

That's a good question, and nothing I can easily answer. I would see it similar to "are modules plugins?". This is also something @abadger wanted to look at.

Yeah, the way that the ansible-core code works, modules really are a type of plugin. roles are not. I'm working up an alternate structure for felixfontein's code to see if it looks like it will be easier to maintain as a separate code path rather than pretending like they're a different type of plugin. I'm not sure yet if it will be... have to see them side by side to decide.

acozine · 2021-07-12T19:41:59Z

tests/functional/schema/good_data/one_role.json

+                    "Felix Fontein (@felixfontein)"
+                ],
+                "description": [
+                    "This is a role which can use any CA supporting the ACME protocol, such as L(Let's Encrypt,https://letsencrypt.org/), L(Buypass,https://www.buypass.com/ssl/products/acme>) or L(ZeroSSL,https://zerossl.com/features/acme/>), to rekey ACME account keys.",


Nice! Way to slip the L(label,link) syntax into the test!

Co-authored-by: Alicia Cozine <[email protected]>

samccann

LGTM

samccann · 2021-07-27T19:47:19Z

Once @acozine gives it a thumbs up we should be good to go! Thanks for all the work on this one.

samccann · 2021-08-03T20:29:34Z

Generated the docs locally and it all looks good!

felixfontein · 2021-08-03T20:33:07Z

@Shrews @wbclark @abadger @briantist @samccann @acozine thanks a lot for your reviews, testing and feedback! :)

felixfontein force-pushed the roles branch 3 times, most recently from 2e4b703 to 00dec46 Compare May 24, 2021 22:24

felixfontein mentioned this pull request May 25, 2021

DaWGs (Documentation Working Group) Meeting Agenda 2021 ansible/community#579

Closed

felixfontein force-pushed the roles branch from 00dec46 to 4b4fb03 Compare May 25, 2021 16:30

felixfontein changed the title ~~[WIP] Generate role documentation~~ Generate role documentation May 25, 2021

felixfontein mentioned this pull request May 26, 2021

The Bullhorn - contribute content for developer newsletter ansible/community#546

Closed

evgeni mentioned this pull request May 27, 2021

use modern role documentation theforeman/foreman-ansible-modules#1235

Open

Shrews reviewed May 28, 2021

View reviewed changes

felixfontein force-pushed the roles branch from 4b4fb03 to b2e422d Compare June 1, 2021 18:40

felixfontein requested a review from abadger June 5, 2021 13:15

felixfontein mentioned this pull request Jun 8, 2021

[WIP] Add new semantic markup support to antsibull-docs #281

Closed

felixfontein force-pushed the roles branch from 8e92ab3 to d326797 Compare June 19, 2021 06:13

felixfontein mentioned this pull request Jun 22, 2021

Create devel docs #282

Merged

felixfontein force-pushed the roles branch from d326797 to e9bdeb7 Compare June 22, 2021 18:36

abadger reviewed Jun 22, 2021

View reviewed changes

antsibull/docs_parsing/ansible_doc.py Outdated Show resolved Hide resolved

briantist reviewed Jul 8, 2021

View reviewed changes

antsibull/data/docsite/role.rst.j2 Outdated Show resolved Hide resolved

samccann reviewed Jul 8, 2021

View reviewed changes

acozine reviewed Jul 12, 2021

View reviewed changes

felixfontein force-pushed the roles branch from 43667b2 to 0ad6f18 Compare July 12, 2021 20:38

Read and generate role documentation.

9a7e957

Co-authored-by: Alicia Cozine <[email protected]>

felixfontein force-pushed the roles branch from 0ad6f18 to 9a7e957 Compare July 12, 2021 21:03

samccann approved these changes Jul 27, 2021

View reviewed changes

felixfontein requested a review from acozine July 31, 2021 06:48

samccann mentioned this pull request Aug 3, 2021

Consider adding roles to what is catalogued within a collection ansible/ansible-navigator#231

Closed

abadger merged commit f009a2f into ansible-community:main Aug 3, 2021

felixfontein deleted the roles branch August 3, 2021 20:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate role documentation #272

Generate role documentation #272

felixfontein commented May 21, 2021 •

edited

Loading

felixfontein commented May 25, 2021

Shrews May 28, 2021

felixfontein May 28, 2021

felixfontein May 28, 2021

wbclark commented Jun 8, 2021 •

edited

Loading

felixfontein commented Jun 8, 2021

samccann left a comment

samccann Jul 8, 2021

felixfontein Jul 12, 2021

samccann Jul 8, 2021

felixfontein Jul 12, 2021

acozine left a comment

acozine Jul 12, 2021

felixfontein Jul 12, 2021

acozine Jul 12, 2021

abadger Jul 13, 2021

acozine Jul 12, 2021

felixfontein Jul 12, 2021

acozine Jul 12, 2021

felixfontein Jul 12, 2021

acozine Jul 12, 2021

felixfontein Jul 12, 2021

abadger Jul 13, 2021

acozine Jul 12, 2021

samccann left a comment

samccann commented Jul 27, 2021

samccann commented Aug 3, 2021

felixfontein commented Aug 3, 2021

Generate role documentation #272

Generate role documentation #272

Conversation

felixfontein commented May 21, 2021 • edited Loading

felixfontein commented May 25, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wbclark commented Jun 8, 2021 • edited Loading

felixfontein commented Jun 8, 2021

samccann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acozine left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samccann left a comment

Choose a reason for hiding this comment

samccann commented Jul 27, 2021

samccann commented Aug 3, 2021

felixfontein commented Aug 3, 2021

felixfontein commented May 21, 2021 •

edited

Loading

wbclark commented Jun 8, 2021 •

edited

Loading