Fixes for compatibility with Python 3.13 #1321
Conversation
| | 0.838545 0.247025 | ||
| | 0.838697 0.436220 | ||
| | 0.309496 0.833591 | ||
| | 0.629452 0.586355 |
There was a problem hiding this comment.
In docstrings, use 4 spaces instead of a tab character.
.github/workflows/ubuntu.yml
Outdated
| strategy: | ||
| matrix: | ||
| python-version: ['3.12', '3.11', '3.8', '3.9', '3.10'] | ||
| python-version: ['3.13', '3.12', '3.11', '3.8', '3.9', '3.8'] |
There was a problem hiding this comment.
Should keep 3.8? Maybe 3.8, 3.10, 3.12 and 3.13 would be enough?
There was a problem hiding this comment.
OK, so ['3.13', '3.12', '3.11', '3.10', '3.9'] in all the workflows?
| for name in dir(self): | ||
| if name.startswith(prefix): | ||
| function = getattr(self, name) | ||
| if not hasattr(function, "__call__"): |
There was a problem hiding this comment.
Now, NoneType has a docstring, so we need to check that function is indeed a function.
| # and re.VERBOSE (readable regular expressions | ||
| ((?:.|\n)*?) | ||
| ^\s+([>#SX])>[ ](.*) # test-code indicator | ||
| ^\s*([>#SX])>[ ](.*) # test-code indicator |
There was a problem hiding this comment.
In Python 3.13, doctrings are trimmed of "margin" spaces.
|
|
||
| # Remove leading <dl>...</dl> | ||
| # doc = DL_RE.sub("", doc) | ||
| doc = filter_comments(doc) # .strip("\s") |
There was a problem hiding this comment.
I do not remember the reason of the r"\s" strip, but it breaks some tests now.
| # Mismatched number of output lines, and we don't have "..." | ||
| return False | ||
|
|
||
| # Python 3.13 replaces tabs by a single space in docstrings. |
There was a problem hiding this comment.
Python 3.13 converts tab characters in docstrings to simple spaces. This "hack" helps to keep the doctests compatible with this change.
|
Note: Supporting 3.13 may push us into using a newer pip that disallows the Keep in mind, getting everything working on 3.13 where we can release might be a bigger undertaking than just getting the Mathics Kernel adjusted. |
|
OK, this is just to adjust the kernel compatibility, and does not breaks the compatibility with previous versions. |
And that means we cannot do a full release like we just did until everything is fixed. |
.github/workflows/ubuntu.yml
Outdated
| strategy: | ||
| matrix: | ||
| python-version: ['3.13', '3.12', '3.11', '3.8', '3.9', '3.8'] | ||
| python-version: ['3.13', '3.12', '3.10', '3.8'] |
There was a problem hiding this comment.
Did you mean to change 3.8 to 3.9?
There was a problem hiding this comment.
I was saying to keep the even versions, plus the last one (if odd).
There was a problem hiding this comment.
3.8 is the version supported by the last version of Pyston.
There was a problem hiding this comment.
I think we should drop Pyston then in its standalone form. There is still a pyston-lite plugin.
I don't see a reason to drop versions that end in an odd number or not test them.
Python does not say that minor numbers that end in odd numbers have any less status than those with even numbers. Dropping 3.8 makes sense because the Python language drifts and it is extra effort to support older Pythons.
There was a problem hiding this comment.
It is a pitty because Pyston is much more faster than CPython and Pyston-lite, but OK. In any case, we still are compatible with 3.8
mmatera
force-pushed
the
fixes_for_3_13
branch
from
9e2a614 to
f73364b
Compare
|
I don't have a problem with the idea or the code, but the timing of doing this right now is weird. I've just spent the last 3 days if not longer trying to get out a release ensuring all the parts work together. (And I am not sure they completely do). Something like this now ensures things will be a little bit out of whack again and puts us back in the state of a little off-ness. I don't think some of the code will be able to handle 3.13 unless an older pip is used. And code in other repositories may need additional adjusting too. And of course, there are other PRs in the backlog. These change the API which will also require changing some other bits of code outside of this repository. To accommodate moving forward which we need to do and keep some stability for a little while right now, I suggest we have an "unstable" branch that we merge to, along with the master branch which is compatible across other projects. Your thoughts? |
|
Hmm. We could split this up into two parts, isolate that version testing change part and everything else. So that when we are ready to switch over everything there is a small bit of code in this repository that does that. |
As with much of the other stuff I did, the timing depends more on finding the time to do it than on achieving a deadline or a particular general goal.
The changes here (except for the tests) ensure that mathics-core tests still pass in Python 3.13. This is not a release or something that breaks the rest of the code. The changes do not seem to affect the behavior of the rest of the components, but in six months or a year, when we want to start the next release, they would make the process simpler.
That sounds reasonable.
|
This is a problem that can be solved by working in a more coordinated way. From the project's perspective, it is better to have a plan for moving forward that we agree on and discuss. You put in a checklist of ideas for the 8.0.0 release. Some of these have been deferred. Items in that list were picked precisely not to be disruptive. And we have open issues. I suppose we should add upgrading to 3.13 to that list, but 3.13 was not on that list. Upgrading to a newer Python release, especially with the way Python and its ecosystem move nowadays, is disruptive. And if you have a question about doing something you could just mention it or ask about it in advance. |
OK, but the changes in this PR are closely related to the work done before the release, relative to adjusting the BTW, the support for LaTeX expressions in the online Mathics-Django using MathJax, something that was in the list for the pre-release, is done: Mathics3/mathics-django#221 |
You may see it that way, but as the discussion I hope shows, it isn't like that. There could be a lesson here: mention first, code later.
Again, it is not about whether certain changes are good, but more about coordinating, timing, and appropriateness for the situation right now.
Thank you!
Sorry, I lost track of this. In my last look, you mentioned that MathJaX needed to be updated. I see that's been done now. Looking at it and trying now. Thanks for doing this. As for release, We can always come out with a minor update for the Django portion. And that might be a good thing to do. The Django portion has this weirdness that we are not using PyPI for CI testing. It might have something to do with where it is expecting JSON tables to be located or that the tables are not getting created. I let this slide in the overall 8.0.0 revision of everything because things were just getting delayed. |
…le (#1322) `Builtin.get_functions` collects attributes of the subclasses with names starting with a prefix and it assumes that these are methods. Until now, the way to check that they are indeed methods was to check if they have a docstring. What we want to check is whether the method is callable. In Python 3.13, this is needed. This PR implements a callable check.
In the current (master) documentation/doctest implementation, it is assumed that all the docstrings have a "margin" of spaces, from the class indentation. Then, the regular expression looking for doctests fails to find them if there is not at least a single space at the beginning of the line. Python 3.13 removes the "left margin" of docstrings, so the regular expression will fail to detect some tests. This PR changes the regular expression to allow doctests which do not start with a space. Also, in future versions of Python, tab characters in docstrings will be converted into spaces, which will make to fail some tests. In this PR, tabs in doctests are replaced by sequences of 4 spaces, making the tests more robust under this kind of change.
…#1327) This is another pass through the documentation system. The PR includes * Fix inconsistencies in the docstrings (missing '$', escape characters, etc) * Remove the hardcoded code that handles special characters from the `DocumentEntry.latex()`. Almost all special characters are now handled by the mathicsscanner tables. * Increase a little bit the consistency in using the `$` character in docstrings. Now it must be explicitly escaped. * The docstring of `escape_latex` function was improved. * Expressions of the form `'Head'[$var_1$,$var_2$]` now are formatted as expected (`'Head'` as a code variable, and `$var_1$` as a LaTeX expressions). --------- Co-authored-by: R. Bernstein <[email protected]>
Surface bottom added to cone; cone orientation corrected.
Isolate a file open evaluation routine used by `Import`, `OpenRead`, and `OpenWrite`.
Go over docs for `Beta`, `Gamma`, `Product`, and infix operators with no meaning
We want to match the behavior of WMA for `Gamma` and `Product`.
Specifically:
```
Product[k, {k, 3, n}] == n! / 2
Gamma[1+x] == Gamma[1+x]
```
Supercedes #1387
# Introduction
This PR includes support for the following hypergeometric functions:
- `Hypergeometric1F1`
- `HypergeometricPFQ`
- `MeijerG`
- `HypergeometricU`
# Checklist
- [X] ~~Functionality and output (see tests)~~
- [X] Documentation (valid / in line with MMA)
- [X] ~~Function attributes (valid / in line with MMA)~~
- [X] ~~Special cases (valid / in line with MMA)~~
- [X] ~~Special behaviour where `z_?RealQ` leads to a numerical
evaluation (as an earlier MMA alternative to `N[]`)~~
- `z_?MachineNumberQ` is used instead of `RealQ`
- [X] ~~Functions work with `Plot[]`~~
- `Plot[Hypergeometric1F1[3, 2, x], {x, 0.5, 2}]` now works with
`imathics`
- `Plot[HypergeometricU[3, 2, x], {x, 0.5, 2}]` now works with
`imathics`
# Tests
```
== 14 passed, 1 warning in 3.29s ==
```
---------
Co-authored-by: Aravindh Krishnamoorthy <[email protected]>
Co-authored-by: R. Bernstein <[email protected]>
Co-authored-by: Juan Mauricio Matera <[email protected]>
found with the help of `sage --tox -e rst -- mathics/core`
- turn one file into utf8 encoding - fix a bunch of typos found using codespell
just fixing a few more typos and also the lines `variable = variable` that were in 4 files
To handle escape sequences better, such as ignoring them in comments, the branch "revise-escape-sequence-scanning" was started as more major refactor. However, that has become too large and too hard to get right. This PR splits the non-escape sequence portion of the changes. After this, the remaining escape sequence changes will be added back. * .github/workflows/*.yml: Until mini-tweaks branch on mathics-scanner is merged, we need to test this from the mini-tweaks branch * Use TranslateErrorNew, which has more information about the scanner error raised inside the exception * mathics/builtin/messages.py: correct Syntax URL * mathics/core/convert/sympy.py mpypy doesn't like using BasicsSympy as a class type (it is a variable of a class type?) * incomplete() -> get_more_input(). See PR in mathics_scanner for more info * tokeniser.sntx_message now returns the message parameters it used. These are passed along inside the exception handling * add parse_incrementally_by_line() * mathics/docpipeline.py tests now record what section and test number they came from. This does make error testing matching harder, but these are expected to get reduced and converted to pytests. Or use "..." in the doctests. We err on the side of information over vagueness and catering to a misguided testing system. * pyproject.toml: Bump minimum Mathics-Scanner needed. We are making incompatible changes in the Mathics-Scanner project again. * test/builtin/atomic/test_strings2.py: we were (I think) inadvertently unescaping some strings that were meant to be escape sequences * test/core/parser/test_parser.py: test result matches WMA closer
Tracks changes in scanner to handle escape sequences more correctly and
to handle escaped boxing operators more properly.
* .github/workflows: CI testing needs to be done with
revise-escape-sequence-scanning branch
* parser.py: next() needs to catch at least escape sequence errors.
(Maybe more?)
* test_parser.py: we now give a more specific
"NamedCharacterSyntaxError" error
Comment out self.check("x \\\ny", "Times[x, y]") test until we
understand what the right behavior is. (Next time we'll document the
test better.)
- remove parenthese around arguments of for - remove parentheses before = when they are not needed
Use {} instead of None for inital structure cache
Fixes #1405
Add skip_trivial_evaluation. This is also used in the Mathics Debugger to not stop too many at places that are not of interest. (evaluating a literal string, or a Symbol, etc.)
Down the line when documentation is redone, there could be more. For now, I am adding the one that I personally find useful in using the existing docs.
When tracing an evaluation, allow the wrapper function to replace or alter the function retutn value. This is used in the Trepan DebugEvaluate debugger, but it can be used by other tracing tools as well.
Use Python version 3.13 in testing. It's faster
Make use of the ".value" field in BaseElement objects, especially Expression.to_python(). Recall, the parser creates Python equivalents for certain literal values. In particular, a list of strings. For this, a Python tuple of `str` appears. And this kind of thing happens a lot. And note that in contrast to Mathics3 strings, a Python string can be duplicated. So by using `.value` of strings found in parsing, we are not only saving time in creation by not copying, but also memory. And we save time in `to_python()` access as well. In some cases, like datetime objects, we can just skip `to_python()` and use `.value` instead. This motivation for this came from a desire to do a similar thing for SymPy and NumPy expressions, but that will have to wait. In the parser, list element literals are `tuple` types, whereas `to_python ()` returns `lists`. In the places where a Python list for a literal was expected, we have a Python tuple for the literal. In these cases, the code has been adjusted to allow both tuple and list types. Where certain list-only operations like mutation were needed, tuples are converted to lists in a new variable. Methods in module `mathics.core.atoms` have been put in alphabetical order. A bug in `convert_expression_elements` in setting `.value` was fixed. I suspect it wasn't noticed before because we were not making use of `.value`. Some of the `FilePrint` error messages have been aligned better for WMA's (and this makes the errors clearer, too.)
mathics.builtin.arithmetic: Mark "I" constant and add a "value" property tensors: move evaluation code for Transpose to mathics.eval.tensors. Suggest what might possibly happen by adding NumPy code. Order more methods in alphabetical order. Improve or add some comments. eval_tensors.py: convert from DOS format to Unix (CR's removed)
Start tracking locations in specific expressions to aid error reporting and debugging. We need to take care to avoid bloating memory with positions. Right now, start locations are tagged for FunctionApplyRule, an ExpressionPattern -> Python method. Here, the location is the Python method, or rather, can be extracted from that. So we can reuse the location information that Python is storing anyway. For Expressions and ExpressionPatterns that are read in from Files, Things are more complicated.. This needs to be coordinated with a change in mathics-scanner with the same git branch name.
We no longer use position-tracking branch
Save SymPy Integer vale in Integer() and some lint.
|
@mmatera - thanks for this! |
This PR contains the fixed needed to make Mathics compatible with Python 3.13.