Filtering out directories in the case of flat layout

[codeandfire]

Hi,

The Package Discovery docs page states that

To avoid confusion, file and folder names that are used by popular tools (or that correspond to well-known conventions, such as distributing documentation alongside the project code) are automatically filtered out in the case of flat-layout:

and names such as docs, tests, examples etc. are listed below as names of files/folders which will be filtered out.

There is also a warning note on that page which states the following:

Please have in mind that find_namespace: (setup.cfg), find_namespace_packages() (setup.py) and find (pyproject.toml) will scan all folders that you have in your project directory if you use a flat-layout. If used naïvely, this might result in unwanted files being added to your final wheel.

and goes on to give an example of a project containing a namespace package that has top-level directories such as docs and tests which end up getting installed along with the actual package.

So in summary, if we use a flat layout, as long as we do not enable scanning for namespace packages, Setuptools will filter out directories such as docs, tests, etc., but if we do enable scanning for namespace packages, such directories will end up getting included unless we add some more configuration such as the exclude argument (or move to an src-layout). I also ran some tests to confirm whether the actual behaviour matches that stated in the docs and indeed this is the case.

My question is: what is stopping us from filtering out directories such as docs, tests etc. even in the case of namespace packages? Is this just a feature that hasn't been implemented yet, or is any of this intentional and there is something I am missing here?

[codeandfire]

Just to clarify: I do understand that enabling scanning for namespace packages will include any packages without an __init__.py. And therefore directories like docs, tests, etc. can potentially be namespace packages. However considering that, more often than not, directories with these names will be things that the developer doesn't want to include in their distribution, possibly we should consider filtering them out? And include them only if they have been explicitly included by the developer, i.e. through the include argument?

[abravalheri]

Thank you very much @codeandfire for starting this discussion. I think this topic is related to #3259.

My view on the subject is the following:

• In the future I expect users to rely more and more on auto-discovery
• Manual discovery settings is offered as a escape hatch when the users need more control over the files and don't want to change their project organization.
  • But then users should be aware that with great power there must come great responsibility...
• Adding the filtering out rules by default into the manual discovery settings adds complexity and forces us to take decisions about topics that are not obvious. For example:
  • What happens if a user wants to build a package called changelog or examples or manpages? Those could be valid packages right? But they would be filtered out as false positives...

I think that if we want to improve user experience we could go here with a more pedagogical approach, e.g. identify when the scanning results in multiple top-level packages + the user did not specify a where/include/exclude, and only then raise an informative warning. This is my proposal in #3259 (comment).

[codeandfire]

Hi @abravalheri, thanks very much for your response.

I agree with this direction that you would like to avoid filtering and instead leave it to the user to properly specify the where/include/exclude, and raise a warning otherwise if multiple top-level packages are found. My limited point was that the filtering strategy should be uniform, whether we have regular packages or namespace packages. If the goal is to move towards no filtering at all, then that is perfectly fine as well.

Should I take this to mean that we plan to move to the following workflow of automatic discovery?

1. Flat-layout:
   • Namespace packages disabled: Scan all directories in the project root having an __init__.py file.
   • Namespace packages enabled: Scan all directories in the project root.
     If multiple packages are detected, check if the user has specified an include/exclude and raise the warning otherwise.
2. Src-layout:
   • Namespace packages disabled: Scan all directories in the src directory having an __init__.py file.
   • Namespace packages enabled: Scan all directories in the src directory.
     Again, if multiple packages are detected, check if the user has specified an include/exclude and raise the warning otherwise.

And therefore, in the case where we have flat-layout and namespace packages are enabled, we will nearly always have to specify an include/exclude configuration, even when we have only one top-level package, because we need to explicitly exclude directories such as docs, tests, etc.? While in the other three cases, we should not typically need an include/exclude configuration if we have only one top-level package?

[codeandfire]

EDIT: I have moved this to a separate post.

[codeandfire]

@abravalheri, if I may ask you another somewhat related question in this thread: shouldn't developers, typically, not require to configure the include/exclude options, in the case of an src-layout?

If I have understood the src-layout correctly, the src directory is supposed to contain all files - both source code and any other files (data files) - that you plan to include in your distribution. By that argument, is it not counterintuitive to supply include/exclude logics in the case of the src-layout? What I mean is that if everything required for your distribution is already present in the src directory, then you should not have to customize it further by including and excluding certain things.

There might be special cases ... For example, I recall an example you had suggested for exclude_package_data, which discusses the scenario involving directory-specific marker files. These directory-specific marker files may have to be placed within the src directory, but excluded from the distribution. In such a case perhaps the use of include/exclude (or really just exclude) makes sense. However, in general, I'm not able to imagine a use case where one might want to use include/exclude with the src-layout, even if there are multiple top-level packages ...

Please feel free to correct me in case I have missed understanding anything here.

[abravalheri]

Hi @codeandfire, I will try my best to reply to the questions 😅. I think there might be a problem with the terminology I am currently using and/or the terminology in the docs. We might want to change terms to facilitate the understanding.

• When I talk about automatic discovery, I am referring to the process when the user does not pass any configuration for packages. This means that there is only one possibility for automatic discovery:

  • Namespace packages will always be enabled.
  • Scanning of flat-layouts will always include the "safety filter" for common folders. If more than one top-level package is detected, the build will fail (the idea here is to prevent people from submitting improper packages).
    (src-layout is always considered "safe")

• When I talk about manual discovery, I am referring to the process when the user specifies packages = find/find_namespaces/....

  • In manual discovery the user is in total control. No filtering is automatically added.

I agree with the following:

that you plan to include in your distribution. By that argument, is it not counterintuitive to supply include/exclude logics in the case of the src-layout? What I mean is that if everything required for your distribution is already present in the src directory, then you should not have to customize it further by including and excluding certain things.

Indeed if the user is using the src-layout it does make sense to let setuptools do the job and leave packages totally unspecified.

I see the manual discovery as a escape hatch for the cases when the project layout is too customised and automatic discovery does not work. For the sake of simplicity/consistency it makes sense to me that find/find_namespace/... always allow the user to specify where/include/exclude (it would be too much of an effort to block certain keyword arguments depending on the project layout...)

Do you have a different view or suggestion of how this should work? I would like to keep the decision tree as small as possible (there is already too many choices...). I am also baking on the hope that moving forward people will simply leave the packages= config unspecified and the automatic behaviour will cover 99.99% of the cases.

[codeandfire]

Let me know your views on this!

[abravalheri]

These are my thoughts (sorry for the reply in different order 😝):

we can assume that again, in the vast majority of projects, the package exported by the project will share the same name as that of the project (after some simple transformations like replacing any hyphens in the project name with underscores). Then, we can have automatic discovery search for a package with this name in both the flat-layout and the src-layout.

In fact this was the original proposal I made last year while implementing automatic discovery in #2888, but after some discussion the suggestion is that the feature should go the other way around: missing names are derived from the package names.

In theory we could do both things, but the code for the auto-discovery is already too complex and I want to avoid growing it even more 😢.

maybe we can have a config with an argument, say names, which just directly specifies the required package names?

Umm, it seems like a minor improvement when compared to what is already possible today isn't it? For example, we can achieve more or less the same using:

[tool.setuptools.packages.find]
include = ["mypkg1*", "mypkg2*"]

I understand that not having to deal with globs is easier for user, but would it be worth the trouble of adding more means of configuration and explaining them?

Do you think the existing docs imply that the user has to set both include and exclude at the same time?

[codeandfire]

Hi @abravalheri, sorry for the late response ...

In fact this was the original proposal I made last year while implementing automatic discovery in #2888, but after #2887 the suggestion is that the feature should go the other way around: missing names are derived from the package names. In theory we could do both things, but the code for the auto-discovery is already too complex and I want to avoid growing it even more 😢.

Alright. No problem!

Do you think the existing docs imply that the user has to set both include and exclude at the same time?

Well yes, this is precisely the impression I got! The docs give the following example:

mypkg
├── setup.cfg  # and/or setup.py, pyproject.toml
└── src
    ├── pkg1
    │   └── __init__.py
    ├── pkg2
    │   └── __init__.py
    ├── additional
    │   └── __init__.py
    └── pkg
        └── namespace
            └── __init__.py

and the following configuration to include pkg, pkg1 and pkg2 while excluding additional:

[options]
packages = find:
package_dir =
    =src

[options.packages.find]
where = src
include = pkg*
exclude = additional

which is why I felt that the use of exclude is redundant.

[abravalheri]

Thanks @codeandfire, I will submit a PR fixing it shortly!

[abravalheri]

Here we go: #3420