Skip to content

Reduce syntax error noise by swallowing dedents like indents #27170

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
epage merged 2 commits into from
Jul 24, 2026
Merged

Reduce syntax error noise by swallowing dedents like indents #27170

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4201eb3
Test more indentation cases
epage Jul 24, 2026
94d5ee4
Reduce syntax error noise by swallowing dedents like indents
epage Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 0 additions & 11 deletions .../src/rules/pycodestyle/snapshots/ruff_linter__rules__pycodestyle__tests__E111_E11.py.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,17 +45,6 @@ invalid-syntax: Unexpected indentation
14 | mimetype = 'application/x-directory'
|

invalid-syntax: Expected a statement
--> E11.py:14:1
|
12 | print()
13 | #: E114 E116
14 | mimetype = 'application/x-directory'
| ^
15 | # 'httpd/unix-directory'
16 | create_date = False
|

invalid-syntax: Expected an indented block after `if` statement
--> E11.py:45:1
|
Expand Down
11 changes: 0 additions & 11 deletions .../src/rules/pycodestyle/snapshots/ruff_linter__rules__pycodestyle__tests__E112_E11.py.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,17 +34,6 @@ invalid-syntax: Unexpected indentation
14 | mimetype = 'application/x-directory'
|

invalid-syntax: Expected a statement
--> E11.py:14:1
|
12 | print()
13 | #: E114 E116
14 | mimetype = 'application/x-directory'
| ^
15 | # 'httpd/unix-directory'
16 | create_date = False
|

E112 Expected an indented block
--> E11.py:45:1
|
Expand Down
11 changes: 0 additions & 11 deletions .../src/rules/pycodestyle/snapshots/ruff_linter__rules__pycodestyle__tests__E113_E11.py.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,17 +34,6 @@ invalid-syntax: Unexpected indentation
14 | mimetype = 'application/x-directory'
|

invalid-syntax: Expected a statement
--> E11.py:14:1
|
12 | print()
13 | #: E114 E116
14 | mimetype = 'application/x-directory'
| ^
15 | # 'httpd/unix-directory'
16 | create_date = False
|

invalid-syntax: Expected an indented block after `if` statement
--> E11.py:45:1
|
Expand Down
11 changes: 0 additions & 11 deletions .../src/rules/pycodestyle/snapshots/ruff_linter__rules__pycodestyle__tests__E114_E11.py.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,17 +23,6 @@ invalid-syntax: Unexpected indentation
14 | mimetype = 'application/x-directory'
|

invalid-syntax: Expected a statement
--> E11.py:14:1
|
12 | print()
13 | #: E114 E116
14 | mimetype = 'application/x-directory'
| ^
15 | # 'httpd/unix-directory'
16 | create_date = False
|

E114 Indentation is not a multiple of 4 (comment)
--> E11.py:15:1
|
Expand Down
11 changes: 0 additions & 11 deletions .../src/rules/pycodestyle/snapshots/ruff_linter__rules__pycodestyle__tests__E115_E11.py.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,17 +23,6 @@ invalid-syntax: Unexpected indentation
14 | mimetype = 'application/x-directory'
|

invalid-syntax: Expected a statement
--> E11.py:14:1
|
12 | print()
13 | #: E114 E116
14 | mimetype = 'application/x-directory'
| ^
15 | # 'httpd/unix-directory'
16 | create_date = False
|

E115 Expected an indented block (comment)
--> E11.py:30:1
|
Expand Down
11 changes: 0 additions & 11 deletions .../src/rules/pycodestyle/snapshots/ruff_linter__rules__pycodestyle__tests__E116_E11.py.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,17 +23,6 @@ invalid-syntax: Unexpected indentation
14 | mimetype = 'application/x-directory'
|

invalid-syntax: Expected a statement
--> E11.py:14:1
|
12 | print()
13 | #: E114 E116
14 | mimetype = 'application/x-directory'
| ^
15 | # 'httpd/unix-directory'
16 | create_date = False
|

E116 Unexpected indentation (comment)
--> E11.py:15:1
|
Expand Down
11 changes: 0 additions & 11 deletions .../src/rules/pycodestyle/snapshots/ruff_linter__rules__pycodestyle__tests__E117_E11.py.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,17 +34,6 @@ invalid-syntax: Unexpected indentation
14 | mimetype = 'application/x-directory'
|

invalid-syntax: Expected a statement
--> E11.py:14:1
|
12 | print()
13 | #: E114 E116
14 | mimetype = 'application/x-directory'
| ^
15 | # 'httpd/unix-directory'
16 | create_date = False
|

E117 Over-indented
--> E11.py:39:1
|
Expand Down
10 changes: 0 additions & 10 deletions .../src/rules/pycodestyle/snapshots/ruff_linter__rules__pycodestyle__tests__W191_W19.py.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,6 @@ invalid-syntax: Unexpected indentation
2 | multiline string with tab in it'''
|

invalid-syntax: Expected a statement
--> W19.py:5:1
|
4 | #: W191
5 | if False:
| ^
6 | print # indented with 1 tab
7 | #:
|

W191 Indentation contains tabs
--> W19.py:6:1
|
Expand Down
44 changes: 43 additions & 1 deletion crates/ruff_python_parser/resources/invalid/statements/if_extra_indent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,50 @@
# Improving the recovery would require changing the lexer to emit an extra dedent token after `a + b`.
# On invalid indentation, recover as if the indentation wasn't there
if True:
pass
a + b

pass

a = 10

# Multiple nested unexpected indents.
if True:
before_nested
first_nested
second_nested
after_nested

outside_nested

# A valid compound statement inside recovered indentation.
if True:
before_compound
if condition:
nested_compound
recovered_compound
after_compound

outside_compound

# Multiple independent unexpected-indent regions in the same body.
if True:
before_regions
first_region
middle_region
second_region
after_region

outside_regions

# An independent syntax error inside recovered indentation stays visible.
if True:
before_error
broken(,)
after_error

outside_error

# Outstanding unexpected indents are flushed at EOF.
if True:
before_eof
final_eof
16 changes: 15 additions & 1 deletion crates/ruff_python_parser/src/parser/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -736,6 +736,7 @@ impl<'src> Parser<'src> {
mut parse_element: impl FnMut(&mut Parser<'src>),
) {
let mut progress = ParserProgress::default();
let mut unexpected_indents = 0;

let saved_context = self.recovery_context;
self.recovery_context = self
Expand All @@ -745,7 +746,12 @@ impl<'src> Parser<'src> {
loop {
progress.assert_progressing(self);

if recovery_context_kind.is_list_element(self) {
if 0 < unexpected_indents && self.at(TokenKind::Dedent) {
// Ignore this `Dedent` like we ignored the `Indent`, avoiding extra errors from
// being imbalanced
unexpected_indents -= 1;
self.bump(TokenKind::Dedent);
} else if recovery_context_kind.is_list_element(self) {
parse_element(self);
} else if recovery_context_kind.is_regular_list_terminator(self) {
break;
Expand All @@ -763,6 +769,14 @@ impl<'src> Parser<'src> {
self.current_token_range(),
);

if matches!(
recovery_context_kind,
RecoveryContextKind::ModuleStatements | RecoveryContextKind::BlockStatements
) && self.at(TokenKind::Indent)
{
// For this invalid `Indent`, ensure the matching `Dedent` gets consumed as well
unexpected_indents += 1;
}

@MichaReiser MichaReiser Jul 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit late here, so I don't have a great suggestion. But it feels a bit off to put this specific statement recovery logic into the generic recovery flow

@epage epage Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If nothing else, this is at least giving some ideas of what to dig into to further understand this code.

The best I've come up with so far for keeping the statement-specific logic in the statement code is we treat an indent as a list element for block statements and then have the parse statement deal with the indent and recover on the matching dedent. I've not yet checked to see what error cases this seemingly-innocent lie might cause.

self.bump_any();
}
}
Expand Down
7 changes: 0 additions & 7 deletions ...on_parser/tests/snapshots/invalid_syntax@backslash_continuation_indentation_error.py.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,10 +69,3 @@ Module(
4 | | 2
| |____^ Syntax Error: Unexpected indentation
|


|
3 | \
4 | 2
| ^ Syntax Error: Expected a statement
|
9 changes: 1 addition & 8 deletions crates/ruff_python_parser/tests/snapshots/invalid_syntax@if_stmt_misspelled_elif.py.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,8 @@ Module(
|
3 | elf:
4 | pass
| ^ Syntax Error: Expected a statement
5 | else:
| ^^^^ Syntax Error: Expected a statement
6 | pass
|

Expand Down Expand Up @@ -128,10 +128,3 @@ Module(
6 | pass
| ^^^^ Syntax Error: Unexpected indentation
|


|
5 | else:
6 | pass
| ^ Syntax Error: Expected a statement
|
9 changes: 0 additions & 9 deletions ...uff_python_parser/tests/snapshots/invalid_syntax@re_lexing__fstring_format_spec_1.py.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -480,15 +480,6 @@ Module(
|


|
9 | 'format spec'}
10 |
| ^ Syntax Error: Expected a statement
11 | f'middle {'string':\\\
12 | 'format spec'}
|


|
11 | f'middle {'string':\\\
12 | 'format spec'}
Expand Down
7 changes: 0 additions & 7 deletions ...on_parser/tests/snapshots/invalid_syntax@statements__if_extra_closing_parentheses.py.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,10 +75,3 @@ Module(
3 | pass
| ^^^^ Syntax Error: Unexpected indentation
|


|
2 | if True)):
3 | pass
| ^ Syntax Error: Expected a statement
|
Loading
Loading