Extension system #576

tristanlatr · 2022-05-12T02:39:55Z

Provide a better extension system for pydoctor.

This new extension system breaks some compatibility with the old way of customizing pydoctor with option --system-class because we don't need to subclass the ASTBuilder to include custom AST visitor code. This is why the Twisted build is crashing with this new version. If everything works correctly, the Twisted customizations can be completely removed except the part that adds a link to the latest version of the document we're viewing if it's not the latest version.

The extension system currently support 3 types of customizations:

AST builder visitor extensions
Mixin classes to be applied to Documentable subclasses
Post processors to be run after all modules are built

It does not support support customizing renderers, so the code in ZopeInterfaceClassPage is not part of the extension system, but still built in with pydoctor. This point can probably be taken care of in a later PR.

The extension system rely on composition of multiple visitors into one.
We do not not need to subclass the System class to define an extension, we just need to add a new module under pydoctor.extensions (to be packaged to pydoctor). Though, we currently need to define a system class if we want to add a custom extension.

It now much easier to add more extensions and properly isolate each feature in a different module.

~~Add a new CLI argument: --extension to load a custom extension.~~ Since it's still possible to add extensions with a custom system like the following:

from pydoctor import model
class MySystem(model.System):
    custom_extensions = ['mylib._pydoctor_extension'] # Load addional extensions

It think I wont add this new argument right now. Also because this change does not mean pydoctor have now a stable API, more changes will come down the road, so I'de suggest we keep the customization system rather "private".

Document how to write an extension in the narrative documentation- stop linking to the Twisted customizations be cause it doesn't work this way anymore. See documentation here: https://pydoctor--576.org.readthedocs.build/en/576/customize.html#use-a-custom-system-class

Get rid of attributes currAttr and newAttr and use builder.currentAttr intsead.

This means the extension system is now based on composition of several visitor intead of rely on subclassing. We cannot test the TwistedSystem anymore because it's not compatible with the new system. It should not be an issue because we now support all the Twisted customizations.

The new extension system is now 100% installed.

codecov · 2022-05-12T02:41:55Z

Codecov Report

Merging #576 (79e6404) into master (814db7a) will decrease coverage by 0.04%.
The diff coverage is 93.11%.

@@            Coverage Diff             @@
##           master     #576      +/-   ##
==========================================
- Coverage   90.96%   90.92%   -0.05%     
==========================================
  Files          42       45       +3     
  Lines        7383     7699     +316     
  Branches     1606     1668      +62     
==========================================
+ Hits         6716     7000     +284     
- Misses        403      435      +32     
  Partials      264      264

Impacted Files	Coverage Δ
pydoctor/astutils.py	`80.24% <71.42%> (-10.00%)`	⬇️
pydoctor/visitor.py	`90.83% <90.83%> (ø)`
pydoctor/astbuilder.py	`94.93% <94.73%> (-0.04%)`	⬇️
pydoctor/factory.py	`97.01% <97.01%> (ø)`
pydoctor/extensions/zopeinterface.py	`92.66% <98.03%> (ø)`
pydoctor/extensions/__init__.py	`98.36% <98.36%> (ø)`
pydoctor/extensions/deprecate.py	`100.00% <100.00%> (ø)`
pydoctor/model.py	`93.21% <100.00%> (+0.26%)`	⬆️
pydoctor/options.py	`94.01% <100.00%> (ø)`
pydoctor/templatewriter/pages/__init__.py	`82.28% <100.00%> (-2.66%)`	⬇️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 814db7a...79e6404. Read the comment docs.

…xtension-system

…ons package. Use pydoctor.model.System by default.

pydoctor/extensions/deprecate.py

tristanlatr · 2022-05-12T15:53:18Z

pydoctor/extensions/zopeinterface.py

+    def _handleZopeInterfaceAssignment(self, node: Union[ast.Assign, ast.AnnAssign]) -> None:
+        for target in node.targets if isinstance(node, ast.Assign) else [node.target]:
+            dottedname = astutils.node2dottedname(target) 
+            if dottedname and len(dottedname)==1:
+                # Here, we consider single name assignment only
+                current = self.visitor.builder.current
+                if isinstance(current, model.Class):
+                    self._handleZopeInterfaceAssignmentInClass(
+                        dottedname[0], node.value, node.lineno
+                    )
+                elif isinstance(current, model.Module):
+                    self._handleZopeInterfaceAssignmentInModule(
+                        dottedname[0], node.value, node.lineno
+                    )


This new code is needed because we don't subclass the ASTBuilder class anymore, the only entry point we have now is visit_* and depart_* methods. So we need to re-do some semantical analysis job in order to redirect to the correct _handle* method. But it's OK, it's always better than subclassing.

pydoctor/extensions/zopeinterface.py

… a single object inside one extension. This also improves coverage.

Fix an issue in the visitor extension logic that would prevent to the depart() method from beeing called if an SkipNode() exception was raised inside the main visitor.

pydoctor/extensions/deprecate.py

pydoctor/astutils.py

pydoctor/extensions/__init__.py

pydoctor/model.py

And fix NoseVisitor typo.

docs/source/customize.rst

pydoctor/test/test_twisted_python_deprecate.py

pydoctor/test/test_astbuilder.py

tristanlatr · 2022-05-19T01:11:16Z

This PR is ready for a final review @adiroiban :)

tristanlatr · 2022-05-19T16:35:43Z

pydoctor/extensions/zopeinterface.py

-            return None
-        assert isinstance(cls, ZopeInterfaceClass)
+    def depart_ClassDef(self, node: ast.ClassDef) -> None:
+        cls = self.visitor.builder.current.contents.get(node.name)


Would be good to edit the builder to have something like this instead:

Suggested change

cls = self.visitor.builder.current.contents.get(node.name)

cls = self.visitor.builder.last

I'll merge this PR as is create a follow-up PR to refactor this, it might more complicated than expected.

adiroiban

Hi, I am busy these days and will not have time to do a proper review for this one.

I think that we can merge.
if there are any issues, we can fix them later.

overall this looks a big step in the right direction.

Many thanks!

…xtension-system

…re. The older visitor relied on generic_visit() to be recursive. Where the provided generic_visit() is not recursive anymore, and moreover not called automatically when visiting unknow nodes! So what I'm saying here is that since #576 have been merged, we're not visiting the statements inside the 'orelse' field of Try and If nodes, same goes for 'finalbody' and 'handlers'. This commit fixes that issue. The rationale is now the following: All statements in the 'orelse' block of IF nodes and statements in the except handlers of TRY nodes that would override a name already defined in the main 'body' (or TRY 'orelse' or 'finalbody') are ignored. Meaning that in the context of the code below, 'ssl' would resolve to 'twisted.internet.ssl': try: from twisted.internet import ssl as _ssl except ImportError: ssl = None else: ssl = _ssl

tristanlatr added 6 commits May 10, 2022 23:11

Refactor ModuleVistor.

af20e3d

Get rid of attributes currAttr and newAttr and use builder.currentAttr intsead.

Also get rid of newAttr in zopeinterface

06c57dd

Add general purpose visitor pattern implementation, with extensions.

6a8850e

Add a generic Factory object to create customizable model classes.

5a699e3

Introduce pydoctor.extensions to hold the setup code for each extension.

7353dd5

The new extension system is now 100% installed.

tristanlatr and others added 9 commits May 11, 2022 23:18

Simplifications

142370c

Remove twisted-apidoc from the CI since it crashes for now.

7d97537

Fix pyflakes

1e338b4

fix docstring

c7f5605

Merge branch 'master' into extension-system

31d6eed

Fix test_lunr_index()

1db120f

Merge branch 'extension-system' of github.com:twisted/pydoctor into e…

4c68432

…xtension-system

Move module holding code for the extensions into the pydoctor.extensi…

8f605f3

…ons package. Use pydoctor.model.System by default.

Move custom systems to the test_astbuilder module.

d398c39

tristanlatr requested a review from adiroiban May 12, 2022 15:30

tristanlatr commented May 12, 2022

View reviewed changes

pydoctor/extensions/deprecate.py Outdated Show resolved Hide resolved

tristanlatr commented May 12, 2022

View reviewed changes

pydoctor/extensions/deprecate.py Outdated Show resolved Hide resolved

tristanlatr commented May 12, 2022

View reviewed changes

pydoctor/extensions/zopeinterface.py Outdated Show resolved Hide resolved

tristanlatr and others added 7 commits May 12, 2022 12:00

Fix litle bug that would prevent to define multiple mixin classes for…

9c8ecab

… a single object inside one extension. This also improves coverage.

Simplify zopeinterface mixin classes.

5d84b10

Use depart_ClassDef instead of visit_ClassDef.

d1a8feb

Use depart_FunctionDef instead of visit_FunctionDef.

15b8018

Fix an issue in the visitor extension logic that would prevent to the depart() method from beeing called if an SkipNode() exception was raised inside the main visitor.

Refactor visitor.py to use less code.

27f876a

Fix docstring.

8a2fc68

Remove commented code

8e8c259

tristanlatr commented May 12, 2022

View reviewed changes

pydoctor/extensions/deprecate.py Outdated Show resolved Hide resolved

tristanlatr commented May 12, 2022

View reviewed changes

pydoctor/extensions/deprecate.py Outdated Show resolved Hide resolved

tristanlatr commented May 17, 2022

View reviewed changes

pydoctor/astutils.py Show resolved Hide resolved

tristanlatr commented May 17, 2022

View reviewed changes

pydoctor/extensions/__init__.py Show resolved Hide resolved

tristanlatr commented May 17, 2022

View reviewed changes

pydoctor/model.py Outdated Show resolved Hide resolved

tristanlatr added 7 commits May 18, 2022 18:14

More docs in customize.rst

846d7d0

Add some docs to NodeVisitor

2fb4f8f

Fix missing Package

0573bd1

Better typing

d80f3fa

Rename core registration methods to something not plural.

00c1e9e

And fix NoseVisitor typo.

Make set lists and make some attributes private.

effa3bd

Fix links

c2e4ab6

tristanlatr commented May 18, 2022

View reviewed changes

docs/source/customize.rst Outdated Show resolved Hide resolved

tristanlatr commented May 18, 2022

View reviewed changes

docs/source/customize.rst Outdated Show resolved Hide resolved

tristanlatr commented May 18, 2022

View reviewed changes

docs/source/customize.rst Outdated Show resolved Hide resolved

tristanlatr commented May 19, 2022

View reviewed changes

pydoctor/test/test_twisted_python_deprecate.py Outdated Show resolved Hide resolved

tristanlatr commented May 19, 2022

View reviewed changes

pydoctor/test/test_astbuilder.py Outdated Show resolved Hide resolved

tristanlatr added 2 commits May 18, 2022 21:09

Fix bunch of typos

da5acc0

Merge branch 'master' into extension-system

f4c55cf

tristanlatr requested a review from adiroiban May 19, 2022 01:11

tristanlatr commented May 19, 2022

View reviewed changes

adiroiban approved these changes May 19, 2022

View reviewed changes

tristanlatr added 3 commits May 19, 2022 13:00

Add some documentation to the ASTBuilder.

08b56e5

Merge branch 'extension-system' of github.com:twisted/pydoctor into e…

4228103

…xtension-system

Add more code comments for hard to understand area.

79e6404

tristanlatr merged commit 02984d9 into master May 19, 2022

tristanlatr deleted the extension-system branch May 19, 2022 22:48

tristanlatr restored the extension-system branch May 20, 2022 00:25

tristanlatr mentioned this pull request May 20, 2022

Add performance tests based on cpython build runtime #584

Closed

tristanlatr mentioned this pull request May 24, 2022

Visit Try 'orelse', 'finalbody' and 'handlers' and If 'orelse' #589

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extension system #576

Extension system #576

tristanlatr commented May 12, 2022 •

edited

Loading

codecov bot commented May 12, 2022 •

edited

Loading

tristanlatr May 12, 2022

tristanlatr commented May 19, 2022

tristanlatr May 19, 2022

tristanlatr May 19, 2022

adiroiban left a comment

	cls = self.visitor.builder.current.contents.get(node.name)
	cls = self.visitor.builder.last

Extension system #576

Extension system #576

Conversation

tristanlatr commented May 12, 2022 • edited Loading

codecov bot commented May 12, 2022 • edited Loading

Codecov Report

tristanlatr May 12, 2022

Choose a reason for hiding this comment

tristanlatr commented May 19, 2022

tristanlatr May 19, 2022

Choose a reason for hiding this comment

tristanlatr May 19, 2022

Choose a reason for hiding this comment

adiroiban left a comment

Choose a reason for hiding this comment

tristanlatr commented May 12, 2022 •

edited

Loading

codecov bot commented May 12, 2022 •

edited

Loading