Automated developer docs generation based on test cassettes for interconnected components/systems

[jayvdb]

@coala and @GitMateIO Python repos have a specific structure for their tests, and 100% test coverage.

Typically automated docs are generated from the source code as the entry point into the system.

When there is a high percentage of test coverage, using the tests as the starting point for documentation will allow all important parts of the API and its source code to be reached.

The unit and integration tests should be written to be informative to the reader, as they also need to be maintained. Using them as the basis of the documentation requires writing readable tests that tell stories, accurately, and those stories can then be checked during code review to ensure the test sequence does in fact demonstrate the story being described. Validators will need to do some basic checking of the test code (Note coala requires unittest compatibly, but allows pytest to add extra features).

The docs generator could additionally convert test cassettes into

• component and flow diagrams to show how high level components are interconnected based on typical usage
• asciinema-like animation so the reader can see how real data flows between the interconnected components when performing a high level operation

This system generated by this project would need to work well for IGitt and coala-bears repositories, and work with caveats for coala and gitmate.

[jayvdb]

Some possibly interesting reading

https://summerofcode.withgoogle.com/archive/2017/projects/4630773877440512/
https://github.com/thisch/pytest-sphinx
http://www.sphinx-doc.org/en/stable/ext/doctest.html

[jayvdb]

Maybe can generate swagger and similar documents from the test cassettes, documenting the API as it is used by tests residing outside that API's component, rather than documenting the entire component API including elements never used in tests by other components.

https://summerofcode.withgoogle.com/archive/2017/projects/4787965653942272/

[jayvdb]

sphinx is a logical choice, but I think we should be open to other options.

Any non-Python processor is going to be unlikely, as the unit tests and source docstrings are written in Python, but some crazy individual might use some bridge to read the AST.

In the idea phase, we need to decide what we want as the deliverable.

In most of our repos, tests are very well maintained, and we are happy to invest lots of time improving tests. So basing docs on tests, with some extra notation added to the tests, will be a way to ensure the docs have good information injected into them frequently.

The hard part is how do we convert tests + source code into user-friendly docs.

[jayvdb]

"test-driven documentation" seems to be a thing. Might have some relevant prior art.

[jayvdb]

https://github.com/spyder-ide/docrepr might be worth looking into for the source code docs generation