Revise escape sequence scanning #126
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
@mmatera There is still a lot to do regarding writing tests and ensuring we handle errors and everything correctly. But since this is getting large, this is a heads-up as to what's on the horizon. |
rocky
force-pushed
the
revise-escape-sequence-scanning
branch
3 times, most recently
from
a5b13d0 to
1ab77fa
Compare
Escape sequences other than named characters have been removed from the prescanner and put in the scanner.
handle syntax errors in mathics3-tokens.
Tokenizer.code -> Tokenizer.source_text Tokenizer.incomplete -> Tokenizer.get_more_input
Start to show syntax errors.
In particular errors with octal digits and incomplete named errors. Go over docstrings in escape_sequences.py
and add more tests.
named-characters.yml: \[Mu] is letterlike tokeniser.py: Correct identifier or pattern for those having letterlike escape sequences
and also add Theta to the list of letterlike symbols
Replace .format() with f-strings. Add comments around Symbol pattern. sntx_message() Excpetion now saves name, tag, and args
Not sure how this worked before, but it did.
* "$\" is a thing * Correct EscapeSyntaxError error message * Better Symbol tokenization for things like a\[Mu]1. More in next commit though.
for things like \.78\.79 Imporve comments around DRYing identifier/symbol_name extension
This PR has gotten out of hand in size, we'll break it up into smaller chunks.
NamedChracterSyntax should be a new-style TranslateError self.code -> self.source_text misc sntx_message() fixes. Document better.
rocky
force-pushed
the
revise-escape-sequence-scanning
branch
from
ef9b7c5 to
53b1402
Compare
rocky
force-pushed
the
revise-escape-sequence-scanning
branch
from
53b1402 to
74587cc
Compare
TranslateError, TranslateErrorNew, ScanError now become ScannerError
it should be just a little bit faster (and it is more modern)
Use more direct and simpler error class name that is is more like its other subclassed errors.
Member
Author
|
@mmatera merging needs to be coordinated with Mathics3/mathics-core#1403 since exception handling has been revised. (In a previous PR, you commented on the use of After this PR, we'll still need to handle boxing expressions inside strings. Boxing expression outside strings, I think, works. But I haven't been able to get to galatea to understand what is expected versus not. |
rocky
force-pushed
the
revise-escape-sequence-scanning
branch
from
fa1155d to
2422c60
Compare
An invalid escape sequence inside a string, like "\(a \+\)" is not an error. Instead the sequence the same, e.g "\(a \+\)".
If the escape sequenced in a string can be a boxing construct, then this
is not an error in the escape sequence. Otherwise, it is.
For example
"\(" is not an error in a string while "\g" is.
Yes, this a bit involved. But that's the way WA works.
Also, flatten values in box operators for BOXING_CONSTRUCT_SUFFIXES
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactor the scanner. Remove prescanner. Handling escape sequences is not a separate phase, but instead is integrated into the scanning phase.
Fixes #125