Allow package-data for stubs packages

[cdce8p]

I'm currently working on moving the stubs_uploader from setup.py to pyproject.toml. It's used to build and upload the typeshed stubs packages automatically. While doing so I noticed that setuptools rejects stub package names like requests-stubs for package-data. I know that .pyi and py.typed files are auto-included. Each stubs package also contains a METADATA.toml file which should be included in the distribution. Do to the nature of the project, it would be desirable to use include-package-data = false and instead specify the wanted files directly.

This PR slightly modifies the schema to allow for that. If accepted, it would be great if a new version could be released so I include these changes in setuptools.

https://github.com/typeshed-internal/stub_uploader/blob/cbdc0b85972e476b4dedbd626cd12dadf7dd60a9/stub_uploader/build_wheel.py#L51-L84

[cdce8p]

/CC @abravalheri

[abravalheri]

Sorry for the delay.

I think that this is very reasonable. Can we somehow re-use the existing definition?

https://github.com/abravalheri/validate-pyproject/blob/v0.24.1/src/validate_pyproject/plugins/setuptools.schema.json#L251-L261?

If accepted, it would be great if a new version could be released so I include these changes in setuptools.

Sure, it may take a little bit however, I want to wait for the situation to stabilise.

[cdce8p]

Sorry for the late reply. I somehow lost track of it.

I think that this is very reasonable. Can we somehow re-use the existing definition?

https://github.com/abravalheri/validate-pyproject/blob/v0.24.1/src/validate_pyproject/plugins/setuptools.schema.json#L251-L261?

Unless I'm missing something, it isn't really possible unfortunately. While the package name itself is normalized by setuptools, I don't think the name in the package-data dict is. Thus the best we can probably do is add a separate validation function for it to reduce duplications. See 80a6d7d

[cdce8p]

@abravalheri Did you had a chance to look at this again?

[abravalheri]

Hi @cdce8p, sorry for the delay in getting back to this.

Unless I'm missing something, it isn't really possible unfortunately. While the package name itself is normalized by setuptools, I don't think the name in the package-data dict is. Thus the best we can probably do is add a separate validation function for it to reduce duplications.

I am not sure I follow. I tested the following snippet for package-data and exclude-package-data on top of 4b72449 and the added tests seem to pass as expected¹:

        "anyOf": [
          { "$ref": "#/definitions/package-name" },
          { "const": "*" }
        ]

Note that #/definitions/package-name already includes the *-stubs case, so that part is covered.

As for the python-module-name-relaxed format used in the shared definition, I believe it’s consistent with how we validate package and package-dir, so it should be fine. We could technically abstract this further into a new definition in the JSON Schema to reduce repetition even more, but I haven’t found a good name to abstract such a concept, so I think the snippet above is already a reasonable compromise².

Personally, I’d prefer keeping this logic in the JSON Schema itself rather than introducing another format function. If you’re okay with that, I can push my local copy with these changes.

Footnotes

1. The only error I got testing locally on 3.12 was FAILED tests/test_pre_compile.py::test_invalid_examples_api[store/ruff-unknown.toml-cli_pre_compile] - urllib.error.URLError: <urlopen error [Errno 11] Resource temporarily unavailable> which is unrelated to the schema changes. ↩

2. Maybe python-module-name-relaxed-or-wildcard (based on your suggested function)... It is a big name, but it does a good job. ↩

[cdce8p]

I am not sure I follow. I tested the following snippet for package-data and exclude-package-data on top of 4b72449 and the added tests seem to pass as expected:

        "anyOf": [
          { "$ref": "#/definitions/package-name" },
          { "const": "*" }
        ]

Note that #/definitions/package-name already includes the *-stubs case, so that part is covered.

As for the python-module-name-relaxed format used in the shared definition, I believe it’s consistent with how we validate package and package-dir, so it should be fine.

Not quite, the point I'm worried about is that the package name itself is normalized by setuptools, however the package-data module name isn't. Even with python_module_name_relaxed aborting early for stubs packages, this would still allow invalid names for modules inside the package-data dict. E.g.

[project]
name = "some-project"

[tool.setuptools.package-data]
"some-project" = ["*.yaml"]

Files stored in some_project/.... Validation would succeed while the data files are not included.

[cdce8p]

Any update?