Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RemoveImportsVisitor crashes when parsing Literal["r+"] #924

Open
Hnasar opened this issue May 17, 2023 · 1 comment
Open

RemoveImportsVisitor crashes when parsing Literal["r+"] #924

Hnasar opened this issue May 17, 2023 · 1 comment
Labels
bug Something isn't working codemod Bundled codemods, visitors, metadata providers

Comments

@Hnasar
Copy link

Hnasar commented May 17, 2023

https://peps.python.org/pep-0586/#shortening-unions-of-literals

The parser is trying to see if the string "r+" refers to some stringified-type and fails to parse it as an expression, but it should just take "r+" as a literal string.

This crashes in both the python and rust/native parser as of libcst 0.4.9

Repro:

# foo.py
from typing import Literal
x: Literal["r+"] = "r+"
# mods/mymod.py
import libcst as cst
from libcst.codemod import VisitorBasedCodemodCommand
from libcst.codemod.visitors import RemoveImportsVisitor


class CrashMod(VisitorBasedCodemodCommand):
    DESCRIPTION: str = "Crash on literal"

    def leave_AnnAssign(
        self, original_node: cst.AnnAssign, updated_node: cst.AnnAssign,
    ) -> cst.AnnAssign:
        RemoveImportsVisitor.remove_unused_import(self.context, "typing")
        return updated_node
# .libcst.codemod.yaml
generated_code_marker: '@generated'
formatter: ['black', '-']
blacklist_patterns: []
modules:
- 'mods'
repo_root: '.'

Output:

python -m libcst.tool codemod mymod.CrashMod foo.py
...
  File "libcst/codemod/visitors/_gather_string_annotation_names.py", line 65, in visit_SimpleString
    self.handle_any_string(node)
  File "libcst/codemod/visitors/_gather_string_annotation_names.py", line 74, in handle_any_string
    mod = cst.parse_module(value)
  File "libcst/_parser/entrypoints.py", line 109, in parse_module
    result = _parse(
  File "libcst/_parser/entrypoints.py", line 56, in _parse
    return _pure_python_parse(
  File "libcst/_parser/entrypoints.py", line 89, in _pure_python_parse
    result = parser.parse()
  File "libcst/_parser/base_parser.py", line 110, in parse
    self._add_token(token)
  File "libcst/_parser/base_parser.py", line 186, in _add_token
    raise ParserSyntaxError(
libcst._exceptions.ParserSyntaxError: Syntax Error @ 1:3.
Incomplete input. Unexpectedly encountered '\n'.

r+
  ^
@zsol zsol added bug Something isn't working codemod Bundled codemods, visitors, metadata providers labels May 25, 2023
@Hnasar
Copy link
Author

Hnasar commented Jul 18, 2023

This also fails when parsing Literal["1d", "1w"]

  File foo line 635, in transform_module
    mod = RemoveImportsVisitor(self.context).transform_module(mod)
  File "venv3.10/lib/python3.10/site-packages/libcst/codemod/_codemod.py", line 108, in transform_module
    return self.transform_module_impl(tree_with_metadata)
  File "venv3.10/lib/python3.10/site-packages/libcst/codemod/_visitor.py", line 32, in transform_module_impl
    return tree.visit(self)
  File "venv3.10/lib/python3.10/site-packages/libcst/_nodes/module.py", line 90, in visit
    result = super(Module, self).visit(visitor)
  ...
  File "venv3.10/lib/python3.10/site-packages/libcst/_nodes/base.py", line 219, in visit
    should_visit_children = visitor.on_visit(self)
  File "venv3.10/lib/python3.10/site-packages/libcst/matchers/_visitors.py", line 720, in on_visit
    return CSTVisitor.on_visit(self, node)
  File "venv3.10/lib/python3.10/site-packages/libcst/_visitors.py", line 123, in on_visit
    retval = visit_func(node)
  File "venv3.10/lib/python3.10/site-packages/libcst/codemod/visitors/_gather_unused_imports.py", line 71, in visit_Module
    node.visit(annotation_visitor)
  File "venv3.10/lib/python3.10/site-packages/libcst/_nodes/module.py", line 90, in visit
    result = super(Module, self).visit(visitor)
  ...
  File "venv3.10/lib/python3.10/site-packages/libcst/_nodes/base.py", line 228, in visit
    _CSTNodeSelfT, self._visit_and_replace_children(visitor)
  File "venv3.10/lib/python3.10/site-packages/libcst/_nodes/expression.py", line 1460, in _visit_and_replace_children
    value=visit_required(self, "value", self.value, visitor),
  File "venv3.10/lib/python3.10/site-packages/libcst/_nodes/internal.py", line 81, in visit_required
    result = node.visit(visitor)
  File "venv3.10/lib/python3.10/site-packages/libcst/_nodes/base.py", line 219, in visit
    should_visit_children = visitor.on_visit(self)
  File "venv3.10/lib/python3.10/site-packages/libcst/matchers/_visitors.py", line 720, in on_visit
    return CSTVisitor.on_visit(self, node)
  File "venv3.10/lib/python3.10/site-packages/libcst/_visitors.py", line 123, in on_visit
    retval = visit_func(node)
  File "venv3.10/lib/python3.10/site-packages/libcst/codemod/visitors/_gather_string_annotation_names.py", line 65, in visit_SimpleString
    self.handle_any_string(node)
  File "venv3.10/lib/python3.10/site-packages/libcst/codemod/visitors/_gather_string_annotation_names.py", line 74, in handle_any_string
    mod = cst.parse_module(value)
  File "venv3.10/lib/python3.10/site-packages/libcst/_parser/entrypoints.py", line 109, in parse_module
    result = _parse(
  File "venv3.10/lib/python3.10/site-packages/libcst/_parser/entrypoints.py", line 55, in _parse
    return parse(source_str)
libcst._exceptions.ParserSyntaxError: Syntax Error @ 1:1.
parser error: error at 1:2: expected one of !=, %, &, (, *, **, +, ,, -, ., /, //, ;, <, <<, <=, ==, >, >=, >>, @, NEWLINE, [, ^, and, if, in, is, not, or, |

1d
^```

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working codemod Bundled codemods, visitors, metadata providers
Projects
None yet
Development

No branches or pull requests

2 participants