m4/pyproject_toml_metadata.m4: Allow Python 3.12

[mkoeppe]

We allow Python 3.12 in the distribution packages of the Sage library, based on the downstream reports by @tornaria and @kiwifb

After parts of this was done in other PRs in the meantime, this amounts to adding the missing classifier
Programming Language :: Python :: 3.12 in a bunch of files, including m4/pyproject_toml_metadata.m4.

📝 Checklist

• The title is concise, informative, and self-explanatory.
• The description explains in detail what this PR is about.
• I have linked a relevant issue or discussion.
• I have created tests covering the changes.
• I have updated the documentation accordingly.

⌛ Dependencies

[mkoeppe]

Someone please set #36561 back to "positive review"

[dimpase]

This is a nightmare. We have 10 places where you by hand have to fill 3.12 (or 3.13).

This is m4. m4 I suppose is capable of getting LT_VERSION or LE_VERSION or whatever from some central place,
like m4/version.m4 or something.

[mkoeppe]

This is a nightmare. We have 10 places where you [...] have to fill [...]

That's right, Dima! But you are phrasing this rather ambiguously.

Every Python distribution package has standard metadata, which includes the python_requires information and version constraints of dependencies.

In a multirepo layout, as discussed at length in #36777, each distribution package in its separate GitHub repository would require a separate pull request to update their metadata.

But thanks to our advanced monorepo tooling (./bootstrap + m4/ + the package information organized in builds/pkgs), we get away with making the change only in m4/pyproject_toml_metadata.m4 --- except for the packages that have not been converted yet from setup.cfg to the modern metadata format pyproject.toml; see #36561.

[tornaria]

I am having a couple of issues here:
a. sage-setup is pinned to 10.3.b1, but I have 10.2 installed in my system. Let me tell you first that packaging sage-setup from pypi as pep517 was a breeze. But if you want me to package and use sage-setup, you need to be more lenient here.
b. sage-conf is required. No, it's not required. It's not required. I tell you it's not required. I need a couple of lines which I handle with a simple sage_conf.py file, but I'm close to not requiring anything there. IMO we should try be reasonable on what we expect from distros and work without sage_conf at all.

Under some circumstances I ended up with a distribution (wheel) named UNKNOWN. I'm thinking this may be be because I had not run bootstrap or something (I sure did run bootstrap, but maybe I cleaned up without realizing). I don't know what can be done about this, but at least pyproject.toml file seems like it should be a static file, not m4. If a particular version of sage-setup is necessary (because a new feature, say) this should be tuned by hand.

In an ideal world, sage-setup has an independent version number with independent releases, and sagemath-standard doesn't require a particular sage-setup version that is not released (this means: not even in beta, because if I'm going down the road of packaging sage-setup I need to be sure I'll be able to build betas without having to muck with the sage-setup that comes with the distro).

My workflow here is something like

1. ./bootstrap sagelib
2. cd pkgs/sagemath-standard
3. python -m build --no-isolation --wheel

I'm still unsure how this will work out in the end... I'm also trying cd src in step 2 to compare. This is taking long because, for some reason, building in this way seems to be giving all misses in ccache, even if I should be compiling the same files again and again. Do you have any clue about this?

[mkoeppe]

I don't know what can be done about this, but at least pyproject.toml file seems like it should be a static file, not m4.

It's m4 so that we can have central synchronization of version info and shared boilerplate.

[mkoeppe]

In an ideal world, sage-setup has an independent version number with independent releases, and sagemath-standard doesn't require a particular sage-setup version that is not released

Yes, in an ideal world, but I'm not going to do this while the modularization work is still ongoing.

Right now, all versions are synchronized because otherwise we would have to test compatibility with other published versions. I don't have the bandwidth for this.

[mkoeppe]

This is taking long because, for some reason, building in this way seems to be giving all misses in ccache, even if I should be compiling the same files again and again. Do you have any clue about this?

python -m build uses a temporary directory for the build, and there is no option to change that. That's

• ccache for Python packages built in temporary directories (help wanted) #35874

[mkoeppe]

Creating a ccache-friendly version of pypa/build would be worthwhile

[kiwifb]

I do work from git checkout for develop and the tip of Volker's merging tree. For releases, I just use pypi sdist.
When using checkout, and I do this for

• sage_conf_pypi (patched with my own values and to not run configure)
• sage_setup
• sage_docbuild
• sage [soon to be renamed sagemath-standard in sage-on-gentoo because that is really what it is now]
• sagemath-bliss
• sagemath-meataxe
• sagemath-sirocco
  I have high hopes of using sagemath-doc-html and sagemath-doc-pdf in the near future.
  I have boiler plate that does
• git checkout of the sage tree
• run bootstrap
• run python -m build -n -x -s pkg/${pkgname}
• untar the resulting sdist
  Next comes any distro patching I carry, then I do a regular wheel build and install with gpep517.
  My main dev machine runs a sage from Volker's merging tree and I have individual packages that do install in sync.

[mkoeppe]

• sage_conf_pypi (patched with my own values and to not run configure)

Would something like sage-conf_static be useful to reduce the amount of patching?

[kiwifb]

• sage_conf_pypi (patched with my own values and to not run configure)

Would something like sage-conf_static be useful to reduce the amount of patching?

Have to look it up. I have not seen it yet, but one of my requirement is for it to be on pypi at release time. It is annoying if it isn't.

[mkoeppe]

one of my requirement is for it to be on pypi at release time.

OK, no, it wouldn't be on PyPI. There can only be one version of sage-conf on PyPI -- the one for installing from PyPI

[kiwifb]

I misunderstood your sentence. I thought you meant something existing. Yes, It could be interesting, so long I can get a sdist from pypi at release time.

[kiwifb]

one of my requirement is for it to be on pypi at release time.

OK, no, it wouldn't be on PyPI. There can only be one version of sage-conf on PyPI -- the one for installing from PyPI

That's it for that. Nevermind, it is not that onerous. Possibly a build option?

[mkoeppe]

Possibly a build option?

Yes, we can look into doing that.

[kiwifb]

Possibly a build option?

Yes, we can look into doing that.

Mark as future work for when I have time (which is technically never :) )

[tornaria]

In an ideal world, sage-setup has an independent version number with independent releases, and sagemath-standard doesn't require a particular sage-setup version that is not released

Yes, in an ideal world, but I'm not going to do this while the modularization work is still ongoing.

Right now, all versions are synchronized because otherwise we would have to test compatibility with other published versions. I don't have the bandwidth for this.

Does sage-setup change that much? What about having a slow release cycle? Even if sage-setup stays in the monorepo, it could have its own release version, and sage-the-distro could install sage-setup from the released version in pypi instead of the version in the monorepo.

Again: if sage-the-distro aims to be a "reference distribution" it should eat its own dogfood, not have "shortcuts".

Also as you explained in #36754 (comment), building sagemath-standard is not tested at all, so... But again, the answer IMO is to always build sage-the-lib using a released version of sage-setup (afaict, the last relevant change to sage_setup was the move to cython 3).

[tornaria]

This is taking long because, for some reason, building in this way seems to be giving all misses in ccache, even if I should be compiling the same files again and again. Do you have any clue about this?

python -m build uses a temporary directory for the build, and there is no option to change that. That's

* [ccache for Python packages built in temporary directories (help wanted) #35874](https://github.com/sagemath/sage/issues/35874)

This would be a blocker for me. Rebuilding sagelib from clean slate takes 1:20 (most of the time cythonizing 562 files, as cython doesn't support ccache afaik). Building gives 0 ccache misses, with 563 direct hits (before cpp) and 63 preprocessed hits.

To be fair, I rerun the pep517 build with the exact same options as the setup.py build, and it took 2:08. The time difference may be in creating the wheel (i.e. zipping, which seems to be single thread). My options (via environment): SAGE_DEBUG=no SAGE_NUM_THREADS=8 PYTHONDONTWRITEBYTECODE=yes. This had 1 miss, 549 direct hits, 76 preprocessed hits. The total is the same 626 compilations.

I wonder if the SAGE_DEBUG=no makes a difference in terms of cacheability (sometimes debug information adds full paths which affect reproducibility).

[mkoeppe]

Under some circumstances I ended up with a distribution (wheel) named UNKNOWN. I'm thinking this may be be because I had not run bootstrap or something (I sure did run bootstrap, but maybe I cleaned up without realizing).

This could be

• bootstrap -s: Configure tarball is missing the files generated from pkgs/*/*.m4 #35268

[mkoeppe]

Does sage-setup change that much?

Yes, at least for a little while until I have consolidated the setup.py's of the modularized distributions in #35095; to be upstreamed in #36649, #36566. Also #36449 just added new stuff to it.

[mkoeppe]

Let's get this in?

[mkoeppe]

Now the last setup.cfg is gone.

[tornaria]

I'm not reviewing 53 commits and 121 files changed just to change 3.11 for 3.12.

I've fixed all the low level stuff so that sage works on python 3.12, and I've been running and testing it on python 3.12 for a couple of months, first on cython 0.29 and now on cython 3. It works, and if someone has python 3.12 installed on their system, just let them use it. In general, stop adding upper bounds on unreleased versions. Since they are unreleased, you don't know that they are incompatible.

TL;DR: Just change the damn number, or kill the upper bound altogether.

[mkoeppe]

I've fixed all the low level stuff so that sage works on python 3.12, and I've been running and testing it on python 3.12 for a couple of months, first on cython 0.29 and now on cython 3.

Yes, thanks, your report that it works is already acknowledged in the PR description. Thanks for all your work on it.

[enriqueartal]

I have tested this branch (merged with #36181, if not the package system was not accepted) in Fedora 39. Only two failed tests, src/sage/libs/giac/__init__.py (this happens also in other instance in the same computer if giac is the system package) and src/sage/libs/gap/element.pyx (only with python 3.12, crash log file attached).
crash_vhg42486.log

[enriqueartal]

The giac failure disappear with the giac package; the other one does not disappear even with the gap package instead of the system package

I have tested this branch (merged with #36181, if not the package system was not accepted) in Fedora 39. Only two failed tests, src/sage/libs/giac/__init__.py (this happens also in other instance in the same computer if giac is the system package) and src/sage/libs/gap/element.pyx (only with python 3.12, crash log file attached). crash_vhg42486.log

[mkoeppe]

Happy new year, and thanks a lot for testing!

Which doctest is failing in src/sage/libs/gap/element.pyx? We have seen sporadic segfaults from this file elsewhere, for example in https://github.com/sagemath/sage/actions/runs/6840579851/job/18600012965#step:11:13016 as documented in https://github.com/sagemath/sage/pull/36937/files#diff-c6591fddad6cbbb6477ab5e8cc342b6c7b89b09f2707c7cc7aa818b52693448dL17

[enriqueartal]

Happy new year! I am not able to isolate the test, maybe you can do it from the above attached crash log. It says only # Killed due to segmentation fault

[enriqueartal]

The giac failure disappear with the giac package; the other one does not disappear even with the gap package instead of the system package

I have tested this branch (merged with #36181, if not the package system was not accepted) in Fedora 39. Only two failed tests, src/sage/libs/giac/__init__.py (this happens also in other instance in the same computer if giac is the system package) and src/sage/libs/gap/element.pyx (only with python 3.12, crash log file attached). crash_vhg42486.log

[mkoeppe]

Happy new year, and thanks a lot for testing!

Which doctest is failing in src/sage/libs/gap/element.pyx? We have seen sporadic segfaults from this file elsewhere, for example in https://github.com/sagemath/sage/actions/runs/6840579851/job/18600012965#step:11:13016 as documented in https://github.com/sagemath/sage/pull/36937/files#diff-c6591fddad6cbbb6477ab5e8cc342b6c7b89b09f2707c7cc7aa818b52693448dL17

[enriqueartal]

Happy new year! I am not able to isolate the test, maybe you can do it from the above attached crash log. It says only # Killed due to segmentation fault

[mkoeppe]

If the crash is reproducible, could you run it with sage -t --verbose?

[enriqueartal]

I attach the output of the test and the crash.

error.txt

crash_9hp8rj9y.log

[mkoeppe]

According to error.txt, the segfault occurs in:

sage: for i in range(100):
    rnd = [ randint(-10,10) for i in range(randint(0,7)) ]
    # compute the sum in GAP
    _ = libgap.Sum(rnd)
    try:
        libgap.Sum(*rnd)
        print('This should have triggered a ValueError')
        print('because Sum needs a list as argument')
    except ValueError:
        pass ## line 2489 ##

FWIW, that's different from the failing run that I linked, https://github.com/sagemath/sage/actions/runs/6840579851/job/18600012965#step:11:13016, which segfaulted in many earlier lines, starting from line 1455.

Edit: That's now #37026

[mkoeppe]

I've added fedora-39 to the systems tested in #36181

[jhpalmieri]

I'm posting to record a vote of -1 on behalf of Tobias Diez.

[kiwifb]

+1 for me.

[mkoeppe]

I'm counting:

• in favor: @mkoeppe @kiwifb
• against: Tobias Diez

Hence I am setting it to "positive review".

[tornaria]

Similar to #36561 and #36951.

-1 from me

[github-actions]

Documentation preview for this PR (built with commit 8fdedff; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

[mkoeppe]

I've rebased it away from its open dependency.

[tornaria]

LGTM

I wonder, since there already is a line requires-python = ">=3.9, <3.13" what is really the role of these "classifiers" strings?

[mkoeppe]

Thanks @tornaria. I don't have a better answer than "this is traditional decoration".

[mkoeppe]

@tornaria Since this PR is "disputed", we have to include the vote tally. I'm assuming that your LGTM means that your -1 vote from #36869 (comment) is replaced by a +1 vote, which brings us to:

• 3 in favor: @mkoeppe @kiwifb @tornaria
• 1 against: @tobiasdiez

Hence the label "positive review" is correct.

[tornaria]

Thanks @tornaria. I don't have a better answer than "this is traditional decoration".

Ok, so it's the metadata that shows up e.g. in pypi (see: https://pypi.org/project/sagemath-standard/). I'm guessing that the requires-python is the "formal" requirement (to be used by tools) and the classifiers are just for human consumption (maybe for search tools).

[mkoeppe]

Well, I think the classifiers are intended to be machine-readable. https://pypi.org/project/trove-classifiers/ is the canonical reference. The design is from over 20 years ago, PEP 301.