Can this be used to just check for compatibility breaks with the __init__.py's __all__ exports?

[matthewdeanmartin]

I may misunderstand the goals of this project, but it looks like it would be useful for checking for breaks in public interface, even thought python doesn't really have anything private except for maybe nested functions.

There is the convention that the __all__ exports from the __init__.py file is the public interface, the objects that the developer would like people to take a dependency on and if you do a deep import or just import anything & that changes, well you get what you deserve.

But griffe seems to assume the radical opposite, that everything that is reachable is part of the public interface, so almost any additional coding would be a semver major change.

So is there a way to, griffe dump my_package.__init__.__all__ or the like?

[pawamoy]

In Python, the convention is that any object that does not start with an underscore _ is considered public. That comprises modules, functions, classes, methods and any attributes. If you want to mark an object as "private" (or "protected", I never remember which one uses name mangling with two leading underscores __), you must prefix it with _.

Griffe also actually supports __all__ attributes. If you define one in a module (any module, not just __init__.py), then Griffe knows that only the names within __all__ are to be considered public, even if other objects not in __all__ do not start with _. If you don't define __all__, then Griffe falls back on whether objects in the module start with _.

If you don't want to prefix most objects with _, you can just prefix their modules. For example:

src/
  package/
    __init__.py
    _mod_a.py
    _mod_b.py

...and then expose just what you want to make public in __init__.py:

from package._mod_a import A
from package._mod_b import B

__all__ = ["A", "B"]

If you don't want to make modules private, you can even prefix the package itself:

src/
  _package/
    __init__.py
    mod_a.py
    mod_b.py
  package/
    __init__.py

# src/package/__init__.py
from _package.mod_a import A
from _package.mod_b import B

__all__ = ["A", "B"]

(...and include both _package and package in your distributions)

With that in mind, I believe Griffe's assumptions are correct and expected. Happy to hear your thoughts 🙂

[matthewdeanmartin]

The _package and using __all__ everywhere are close, I'll try it out starting with using __all__ everywhere, that is the least intrusive.

Last time I worked in a language with real access modifiers you needed public, protected internal, protected, internal, private protected, private, InternalsVisible and "friend", to have usable access modifier semantics. That is a lot of work for a single _ to do.

Pylint treats _ as something on the fairly private end of the spectrum (so that unit tests of the _() are bad, other modules in the same package using _() is bad, etc.) so I usually just get frustrated and disable the rule and if even the linters can't give sensible lint-time restrictions then why add a Hungarian notation wart to every variable? I lost years of cartilage off my hands typing Hungarian notation warts on vb variables and was happy to see the mass rejection of Hungarian notation in almost all modern languages. The _ is the camel poking its hungarian notation nose back under the tent. (Imagine if instead of annotations the mypy people had opted for private_to_package_but_not_unittests_generic_list_of_union_of_strings_and_bytes_names as the mandatory way to express access modifiers and typing.

[matthewdeanmartin]

Man the more I think about the worse Hungarian access modifiers sound. It really feels like a language design mistake that would have been handled by annotations if annotations were a think earlier in python's evolution.

[pawamoy]

I usually just get frustrated and disable the rule

I can totally relate. I prefix names with _ because they are internal and shouldn't be used by others with expectations that their API is stable. But if I use/import them myself, linters keep telling me "it's protected, don't do that"... duh, it's my own stuff, I think I know what I'm doing. It's not that hard to detect that a name is declared within this very package and not warn about its use. But linters have a bad habit of processing code one file at a time, and do not build knowledge about the whole package 😕

I wish there was a proper standard for declaring names as public. As you pointed it out, _ is less than optimal. But here we are, hijacking __all__ when it was originally made for wildcard imports... 🤷

For a positive note: Griffe supports extensions, and could therefore support decorators (or other constructs) explicitly marking objects as public, regardless of their name. Either coming from internal utils, or third-party libraries like https://pypi.org/project/public/. There's another package that even enforces it further, removing the possitibility to access or import an object that isn't public, but I don't remember its name. Could be https://pypi.org/project/export/. Ah, I think I was thinking about https://github.com/frostming/modul!