Add token based `parenthesized_ranges` implementation by denyszhak · Pull Request #21738 · astral-sh/ruff

denyszhak · 2025-12-01T20:18:04Z

Summary

Optimizes parenthesized_range by using token slice partitioning to reduce scanning overhead.

Closes #21510

Test Plan

Added unit tests for parenthesized_range and parentheses_iterator
Added coverage to ensure behavior matches the previous implementation

Acknowledging closed PR #21734 by @FarhanAliRaza for a couple of insights that helped shape the final version.

astral-sh-bot · 2025-12-01T20:30:08Z

Diagnostic diff on typing conformance tests

No changes detected when running ty on typing conformance tests ✅

astral-sh-bot · 2025-12-01T20:31:42Z

`mypy_primer` results

Changes were detected when running on open source projects

scikit-build-core (https://github.com/scikit-build/scikit-build-core)
- src/scikit_build_core/build/_pathutil.py:25:38: error[invalid-argument-type] Argument to function `__new__` is incorrect: Expected `str | PathLike[str]`, found `DirEntry[Path]`
- src/scikit_build_core/build/_pathutil.py:27:24: error[invalid-argument-type] Argument to function `__new__` is incorrect: Expected `str | PathLike[str]`, found `DirEntry[Path]`
- src/scikit_build_core/build/wheel.py:98:20: error[no-matching-overload] No overload of bound method `__init__` matches arguments
- Found 44 diagnostics
+ Found 41 diagnostics

pandas-stubs (https://github.com/pandas-dev/pandas-stubs)
- pandas-stubs/_typing.pyi:1207:16: warning[unused-ignore-comment] Unused blanket `type: ignore` directive
- Found 5815 diagnostics
+ Found 5814 diagnostics

pydantic (https://github.com/pydantic/pydantic)
- pydantic/fields.py:943:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:943:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:983:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:983:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1026:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:1026:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1066:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:1066:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1109:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:1109:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1148:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:1148:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1188:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:1188:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1567:13: error[invalid-argument-type] Argument is incorrect: Expected `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`, found `Top[dict[Unknown, Unknown]] | (((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) & ~Top[dict[Unknown, Unknown]]) | None`
+ pydantic/fields.py:1567:13: error[invalid-argument-type] Argument is incorrect: Expected `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`, found `Top[dict[Unknown, Unknown]] | (((dict[str, Divergent], /) -> None) & ~Top[dict[Unknown, Unknown]]) | None`

No memory usage changes detected ✅

astral-sh-bot · 2025-12-01T20:46:15Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

MichaReiser

Nice, this looks great. I've a few smaller nit comments but it's mostly good to go

crates/ruff_python_parser/src/lib.rs

crates/ruff_python_parser/src/parenthesize.rs

crates/ruff_python_ast_integration_tests/tests/parenthesize_optimized.rs

Co-authored-by: Micha Reiser <micha@reiser.io>

…version

denyszhak · 2025-12-02T21:48:42Z

@MichaReiser this one should be ready for you to review again

MichaReiser

Thanks, this is great.

If you're interested, it would be great if we could rewrite the calls to parenthesized_range in ruff_linter to use your new implementation

denyszhak · 2025-12-03T09:52:25Z

Thanks, this is great.

If you're interested, it would be great if we could rewrite the calls to parenthesized_range in ruff_linter to use your new implementation

I'm definitely interested, let me prepare a PR for this, I will create an issue and link it there

* origin/main: [ty] Improve `@override`, `@final` and Liskov checks in cases where there are multiple reachable definitions (#21767) [ty] Extend `invalid-explicit-override` to also cover properties decorated with `@override` that do not override anything (#21756) [ty] Enable LRU collection for parsed module (#21749) [ty] Support typevar-specialized dynamic types in generic type aliases (#21730) Add token based `parenthesized_ranges` implementation (#21738) [ty] Default-specialization of generic type aliases (#21765) [ty] Suppress false positives when `dataclasses.dataclass(...)(cls)` is called imperatively (#21729) [syntax-error] Default type parameter followed by non-default type parameter (#21657) new module for parsing ranged suppressions (#21441) [ty] `type[T]` is assignable to an inferable typevar (#21766) Fix syntax error false positives for `await` outside functions (#21763) [ty] Improve diagnostics for unsupported comparison operations (#21737) Move `Token`, `TokenKind` and `Tokens` to `ruff-python-ast` (#21760) [ty] Don't confuse multiple occurrences of `typing.Self` when binding bound methods (#21754) Use our org-wide Renovate preset (#21759) Delete `my-script.py` (#21751) [ty] Move `all_members`, and related types/routines, out of `ide_support.rs` (#21695)

* origin/main: [ty] Reachability constraints: minor documentation fixes (#21774) [ty] Fix non-determinism in `ConstraintSet.specialize_constrained` (#21744) [ty] Improve `@override`, `@final` and Liskov checks in cases where there are multiple reachable definitions (#21767) [ty] Extend `invalid-explicit-override` to also cover properties decorated with `@override` that do not override anything (#21756) [ty] Enable LRU collection for parsed module (#21749) [ty] Support typevar-specialized dynamic types in generic type aliases (#21730) Add token based `parenthesized_ranges` implementation (#21738) [ty] Default-specialization of generic type aliases (#21765) [ty] Suppress false positives when `dataclasses.dataclass(...)(cls)` is called imperatively (#21729) [syntax-error] Default type parameter followed by non-default type parameter (#21657)

feat: optimize parenthesized_range

602b7ea

denyszhak requested review from AlexWaygood, MichaReiser, carljm, dcreager, dhruvmanila and sharkdp as code owners December 1, 2025 20:18

AlexWaygood removed their request for review December 1, 2025 20:19

feat: parentheses_iterator fix clippy complaint

f45f7d2

MichaReiser requested changes Dec 2, 2025

View reviewed changes

MichaReiser added the internal An internal refactor or improvement label Dec 2, 2025

denyszhak and others added 5 commits December 2, 2025 11:39

Update crates/ruff_python_parser/src/parenthesize.rs

27bbc66

Co-authored-by: Micha Reiser <micha@reiser.io>

Update crates/ruff_python_parser/src/parenthesize.rs

7ed3e89

Co-authored-by: Micha Reiser <micha@reiser.io>

feat: rollback to the regular TokenKind.is_trivia()

9f2abce

feat: parenthesized_range remove integration tests with the original …

b1156a0

…version

Merge branch 'main' into feat-optimized-parenthesized-range

c5ed330

amyreese requested a review from MichaReiser December 2, 2025 23:18

Move to ruff_python_ast

8df7045

MichaReiser changed the title ~~feat: optimize parenthesized_range~~ Add token based parenthesized_ranges implementation Dec 3, 2025

MichaReiser approved these changes Dec 3, 2025

View reviewed changes

MichaReiser enabled auto-merge (squash) December 3, 2025 08:14

MichaReiser merged commit f4e4229 into astral-sh:main Dec 3, 2025
40 checks passed

denyszhak mentioned this pull request Dec 8, 2025

Update ruff_linter to use optimized version of parenthesized_ranges #21835

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add token based `parenthesized_ranges` implementation#21738

Add token based `parenthesized_ranges` implementation#21738
MichaReiser merged 8 commits intoastral-sh:mainfrom
denyszhak:feat-optimized-parenthesized-range

denyszhak commented Dec 1, 2025

Uh oh!

astral-sh-bot bot commented Dec 1, 2025 •

edited

Loading

Uh oh!

astral-sh-bot bot commented Dec 1, 2025 •

edited

Loading

Uh oh!

astral-sh-bot bot commented Dec 1, 2025 •

edited

Loading

Uh oh!

MichaReiser left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

denyszhak commented Dec 2, 2025

Uh oh!

MichaReiser left a comment

Uh oh!

Uh oh!

denyszhak commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

denyszhak commented Dec 1, 2025

Summary

Test Plan

Uh oh!

astral-sh-bot bot commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Diagnostic diff on typing conformance tests

Uh oh!

astral-sh-bot bot commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mypy_primer results

Uh oh!

astral-sh-bot bot commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ruff-ecosystem results

Linter (stable)

Linter (preview)

Formatter (stable)

Formatter (preview)

Uh oh!

MichaReiser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

denyszhak commented Dec 2, 2025

Uh oh!

MichaReiser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

denyszhak commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

astral-sh-bot bot commented Dec 1, 2025 •

edited

Loading

astral-sh-bot bot commented Dec 1, 2025 •

edited

Loading

`mypy_primer` results

astral-sh-bot bot commented Dec 1, 2025 •

edited

Loading

`ruff-ecosystem` results