RFC: Include python3.def itself in the Rust source

[adamreichold]

All parsing and processing is done in the CI so that the crate itself includes the python3.def file itself instead of creating it on the fly.

Based on the idea voiced in #9 (comment)

I did try to keep the existing semantics in place and just moved things around to a Python script running in the CI. I just chose Python because I assume that the ubuntu-latest image contains a reasonably up-to-date version. (The updated CI workflow is untested, but I used the script locally the produce the definition file committed here.)

One could try using the upstream script to do the parsing and generating, but I think that should be evaluated separately.

[ravenexp]

I agree with this change in principle, but I'd like to see at least some test coverage to ensure that the number of Stable ABI exports is non-decreasing with each definitions update PR. The Rust version had something like that.

You should also update LICENSE to remove all mentions of Python sources.

[adamreichold]

I agree with this change in principle, but I'd like to see at least some test coverage to ensure that the number of Stable ABI exports is non-decreasing with each definitions update PR. The Rust version had something like that.

Added this as an assert whenever the parser script is run.

You should also update LICENSE to remove all mentions of Python sources.

Done.

[adamreichold]

Also note that the automatically created PR will contain a diff of the generated definitions file this way, so we can also review that before merging.

[encukou]

The switch to TOML will happen today or early next week.

[ravenexp]

Ignore this.

[ravenexp]

Looks good to me.

[adamreichold]

The switch to TOML will happen today or early next week.

Thanks for the heads-up! I think we could do import toml then without adding any dependencies to the Rust build scripts. (Or do something simpler, I just mean that we unrestricted w.r.t. our dependency closure.)

[ravenexp]

The switch to TOML will happen today or early next week.

Thanks for updating us on this. I'll delay the merge until the update script migrates to the TOML source.

[encukou]

CPython just switched to TOML.

If you stick with Python for parsing, beware that toml is unmaintained/outdated.
Consider tomli, a little, correct TOML parser, which is being added into the Python 3.11 standard library as tomllib. (Full disclosure: I helped)

If you happen to know another project that needs a python3.def (or could use it), please let me know. I could probably generate it and keep it updated in CPython itself to reduce duplication of effort.

[ravenexp]

If you happen to know another project that needs a python3.def (or could use it), please let me know. I could probably generate it and keep it updated in CPython itself to reduce duplication of effort.

Does this mean that python3.def will be committed to the CPython project repository and we will be able to pull it in as is without generating it ourselves from the TOML definitions?

[encukou]

Yes. CPython has some infrastructure to generate files based on stable_abi.toml, one more shouldn't be too much of a burden (though other devs might disagree).
A possible issue is that it wouldn't be tested in CPython, so we'd rely on you to report any issues.

Python generates and uses python3dll.c instead. (Hm, would you be able to use that by any chance? I know next to nothing about building on Windows...)
edit: The exact formatting is as unstable as stable_abi.toml, it's only better if you can feed it to a C compiler directly.

[ravenexp]

Python generates and uses python3dll.c instead. (Hm, would you be able to use that by any chance? I know next to nothing about building on Windows...) edit: The exact formatting is as unstable as stable_abi.toml, it's only better if you can feed it to a C compiler directly.

No, I think it's specific to the Microsoft C++ compiler, and we also support MinGW and LLVM tools (without Clang). Also we only need the import library, not the DLL file itself.

[ravenexp]

Nice, thanks!

[ravenexp]

I think we can merge this now (?). I'll wait for @messense to take a look.

[adamreichold]

Had to switch to installing python3-pip via apt-get and tomli via pip as ubuntu-latest is 20.04 which does not yet contain the python3-tomli package introduced in 22.04.

[ravenexp]

Had to switch to installing python3-pip via apt-get and tomli via pip as ubuntu-latest is 20.04 which does not yet contain the python3-tomli package introduced in 22.04.

TBH I was surprised that Ubuntu had up to date Python packages, but I'm mostly using official Rust docker images based on Debian Buster, so pip install was my go-to choice.

[messense]

👍