[ty] Skip redundancy checks for sub-union-elements by MichaReiser · Pull Request #22071 · astral-sh/ruff

MichaReiser · 2025-12-19T07:54:14Z

Summary

When adding a sub-union to a union, don't compare its elements to each other, as they are already simplified.

While the is_redundant_with calls are mostly cached, reducing the impact of this change, other calls like is_subtype_of aren't cached and can be skipped with this change.

It turned out that this change is slightly trickier than I thought because we build unions in different modes, and the optimization is only correct if both unions use the same mode.

astral-sh-bot · 2025-12-19T07:55:58Z

Diagnostic diff on typing conformance tests

No changes detected when running ty on typing conformance tests ✅

astral-sh-bot · 2025-12-19T07:57:13Z

`mypy_primer` results

Changes were detected when running on open source projects

mkosi (https://github.com/systemd/mkosi)
+ mkosi/qemu.py:527:13: error[invalid-argument-type] Argument is incorrect: Expected `Sequence[Path | str]`, found `list[Unknown | Path | None | str]`
- Found 97 diagnostics
+ Found 98 diagnostics

pydantic (https://github.com/pydantic/pydantic)
- pydantic/fields.py:943:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:943:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:983:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:983:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1026:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:1026:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1066:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:1066:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1109:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:1109:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1148:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:1148:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1188:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:1188:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1567:13: error[invalid-argument-type] Argument is incorrect: Expected `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`, found `Top[dict[Unknown, Unknown]] | (((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) & ~Top[dict[Unknown, Unknown]]) | None`
+ pydantic/fields.py:1567:13: error[invalid-argument-type] Argument is incorrect: Expected `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`, found `Top[dict[Unknown, Unknown]] | (((dict[str, Divergent], /) -> None) & ~Top[dict[Unknown, Unknown]]) | None`

xarray (https://github.com/pydata/xarray)
- xarray/core/dataarray.py:5744:16: error[invalid-return-type] Return type does not match returned value: expected `T_Xarray@map_blocks`, found `DataArray | Dataset`
+ xarray/core/dataarray.py:5744:16: error[invalid-return-type] Return type does not match returned value: expected `T_Xarray@map_blocks`, found `T_Xarray@map_blocks | DataArray | Dataset`
- xarray/core/dataset.py:8873:16: error[invalid-return-type] Return type does not match returned value: expected `T_Xarray@map_blocks`, found `DataArray | Dataset`
+ xarray/core/dataset.py:8873:16: error[invalid-return-type] Return type does not match returned value: expected `T_Xarray@map_blocks`, found `T_Xarray@map_blocks | DataArray | Dataset`

scikit-build-core (https://github.com/scikit-build/scikit-build-core)
- src/scikit_build_core/build/wheel.py:98:20: error[no-matching-overload] No overload of bound method `__init__` matches arguments
- Found 48 diagnostics
+ Found 47 diagnostics

scikit-learn (https://github.com/scikit-learn/scikit-learn)
- sklearn/externals/array_api_extra/_lib/_at.py:300:17: warning[possibly-missing-attribute] Attribute `dtype` may be missing on object of type `int | Array | tuple[int | slice[Any, Any, Any] | EllipsisType | Array, ...] | slice[Any, Any, Any] | EllipsisType`
+ sklearn/externals/array_api_extra/_lib/_at.py:300:17: warning[possibly-missing-attribute] Attribute `dtype` may be missing on object of type `int | slice[Any, Any, Any] | EllipsisType | Array | tuple[int | slice[Any, Any, Any] | EllipsisType | Array, ...]`
- sklearn/externals/array_api_extra/_lib/_at.py:301:17: warning[possibly-missing-attribute] Attribute `shape` may be missing on object of type `int | Array | tuple[int | slice[Any, Any, Any] | EllipsisType | Array, ...] | slice[Any, Any, Any] | EllipsisType`
+ sklearn/externals/array_api_extra/_lib/_at.py:301:17: warning[possibly-missing-attribute] Attribute `shape` may be missing on object of type `int | slice[Any, Any, Any] | EllipsisType | Array | tuple[int | slice[Any, Any, Any] | EllipsisType | Array, ...]`
- sklearn/externals/array_api_extra/_lib/_at.py:308:25: error[invalid-argument-type] Argument to function `apply_where` is incorrect: Expected `Array`, found `int | Array | tuple[int | slice[Any, Any, Any] | EllipsisType | Array, ...] | slice[Any, Any, Any] | EllipsisType`
+ sklearn/externals/array_api_extra/_lib/_at.py:308:25: error[invalid-argument-type] Argument to function `apply_where` is incorrect: Expected `Array`, found `int | slice[Any, Any, Any] | EllipsisType | Array | tuple[int | slice[Any, Any, Any] | EllipsisType | Array, ...]`

bokeh (https://github.com/bokeh/bokeh)
+ src/bokeh/embed/notebook.py:86:37: error[invalid-argument-type] Argument is incorrect: Expected `Theme | type[FromCurdoc] | None`, found `Theme | FromCurdoc | None`
+ src/bokeh/embed/standalone.py:265:28: error[invalid-argument-type] Argument is incorrect: Expected `Sequence[Model]`, found `Model | Sequence[Model] | dict[str, Model]`
- Found 893 diagnostics
+ Found 895 diagnostics

static-frame (https://github.com/static-frame/static-frame)
- static_frame/core/bus.py:675:16: error[invalid-return-type] Return type does not match returned value: expected `InterGetItemILocReduces[Bus[Any], object_]`, found `InterGetItemILocReduces[Self@iloc | Bus[Any], object_ | Self@iloc]`
+ static_frame/core/bus.py:671:16: error[invalid-return-type] Return type does not match returned value: expected `InterGetItemLocReduces[Bus[Any], object_]`, found `InterGetItemLocReduces[Top[Bus[Any]] | TypeBlocks | Batch | ... omitted 6 union elements, object_]`
+ static_frame/core/bus.py:675:16: error[invalid-return-type] Return type does not match returned value: expected `InterGetItemILocReduces[Bus[Any], object_]`, found `InterGetItemILocReduces[Top[Index[Any]] | TypeBlocks | Top[Bus[Any]] | ... omitted 6 union elements, generic[object]]`
- static_frame/core/series.py:772:16: error[invalid-return-type] Return type does not match returned value: expected `InterGetItemILocReduces[Series[Any, Any], TVDtype@Series]`, found `InterGetItemILocReduces[Top[Index[Any]] | Top[Series[Any, Any]] | TypeBlocks | ... omitted 6 union elements, generic[object]]`
+ static_frame/core/series.py:772:16: error[invalid-return-type] Return type does not match returned value: expected `InterGetItemILocReduces[Series[Any, Any], TVDtype@Series]`, found `InterGetItemILocReduces[Top[Index[Any]] | TypeBlocks | Top[Bus[Any]] | ... omitted 6 union elements, generic[object]]`
- static_frame/core/series.py:4072:16: error[invalid-return-type] Return type does not match returned value: expected `InterGetItemILocReduces[SeriesHE[Any, Any], TVDtype@SeriesHE]`, found `InterGetItemILocReduces[TypeBlocks | Batch | SeriesAssign | ... omitted 6 union elements, generic[object]]`
+ static_frame/core/series.py:4072:16: error[invalid-return-type] Return type does not match returned value: expected `InterGetItemILocReduces[SeriesHE[Any, Any], TVDtype@SeriesHE]`, found `InterGetItemILocReduces[Top[Index[Any]] | TypeBlocks | Top[Bus[Any]] | ... omitted 7 union elements, generic[object]]`
- static_frame/core/yarn.py:418:16: error[invalid-return-type] Return type does not match returned value: expected `InterGetItemILocReduces[Yarn[Any], object_]`, found `InterGetItemILocReduces[Top[Index[Any]] | Top[Series[Any, Any]] | Top[Yarn[Any]] | ... omitted 6 union elements, generic[object]]`
+ static_frame/core/yarn.py:418:16: error[invalid-return-type] Return type does not match returned value: expected `InterGetItemILocReduces[Yarn[Any], object_]`, found `InterGetItemILocReduces[Top[Index[Any]] | TypeBlocks | Top[Bus[Any]] | ... omitted 6 union elements, generic[object]]`
- Found 1843 diagnostics
+ Found 1844 diagnostics

jax (https://github.com/google/jax)
- jax/_src/tree_util.py:302:31: error[invalid-argument-type] Argument to bound method `register_node` is incorrect: Expected `(Hashable, Iterable[object], /) -> T@register_pytree_node`, found `(_AuxData@register_pytree_node, _Children@register_pytree_node, /) -> T@register_pytree_node`
- jax/_src/tree_util.py:305:31: error[invalid-argument-type] Argument to bound method `register_node` is incorrect: Expected `(Hashable, Iterable[object], /) -> T@register_pytree_node`, found `(_AuxData@register_pytree_node, _Children@register_pytree_node, /) -> T@register_pytree_node`
- jax/_src/tree_util.py:308:31: error[invalid-argument-type] Argument to bound method `register_node` is incorrect: Expected `(Hashable, Iterable[object], /) -> T@register_pytree_node`, found `(_AuxData@register_pytree_node, _Children@register_pytree_node, /) -> T@register_pytree_node`
- Found 2802 diagnostics
+ Found 2799 diagnostics

No memory usage changes detected ✅

codspeed-hq · 2025-12-19T08:12:38Z

CodSpeed Performance Report

Merging #22071 will improve performance by 32.09%

_{Comparing micha/union-builder-sub-unions (54347a5) with main (7ac1874)}

Summary

⚡ 3 improvements
✅ 19 untouched
⏩ 30 skipped¹

Benchmarks breakdown

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	WallTime	`altair`	5.2 s	4.6 s	+12.73%
⚡	WallTime	`multithreaded`	1.5 s	1.1 s	+32.09%
⚡	WallTime	`pydantic`	10.4 s	9.9 s	+4.98%

30 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

MichaReiser · 2025-12-19T12:22:41Z

The speed-ups in the ty-benchmark are between 0-6%

MichaReiser · 2025-12-19T12:25:42Z

Not sure how to tell whether there are any new mypy primer hits or if it's just the usual non-deterministic cases

AlexWaygood · 2025-12-19T12:27:36Z

Not sure how to tell whether there are any new mypy primer hits or if it's just the usual non-deterministic cases

It might be worth seeing if you can reproduce the prefect errors locally. I don't recall seeing those ones before.

MichaReiser · 2025-12-19T13:09:35Z

Yeah, they look suspicious.

The optimization in PR astral-sh#22071 skips redundancy checks between elements of a sub-union when adding it to a UnionBuilder, assuming the sub-union is already simplified. However, unions created via `from_elements_cycle_recovery` are built with `cycle_recovery=true` (which skips redundancy checks) but `recursively_defined=No` (the default). When such a union is later added to a normal builder: - `!self.cycle_recovery` = true - `self.recursively_defined == union.recursively_defined()` = true (both No) - The optimization kicks in and skips redundancy checks - But the elements were never checked against each other! This caused type inference errors where redundant types remained in unions, leading to `invalid-argument-type` errors for generic type parameters. The fix: Set `recursively_defined=Yes` for unions built during cycle_recovery, so the optimization condition fails when adding them to normal builders, ensuring proper redundancy checks are performed.

MichaReiser · 2026-01-21T13:41:27Z

I don't see getting time to look at this anytime soon, so I'll close the PR but I'll open a follow up issue, because the improvement on seems promising.

MichaReiser added the ty Multi-file analysis & type inference label Dec 19, 2025

MichaReiser marked this pull request as ready for review December 19, 2025 12:24

MichaReiser requested review from AlexWaygood, carljm, dcreager and sharkdp as code owners December 19, 2025 12:24

AlexWaygood added the performance Potential performance improvement label Dec 19, 2025

zanieb mentioned this pull request Dec 19, 2025

[ty] Fix bug in union builder optimization #22085

Closed

MichaReiser marked this pull request as draft December 22, 2025 14:06

MichaReiser added 3 commits December 25, 2025 16:51

[ty] Skip redundancy checks for sub-union-elements

49d16c2

Apply same check for literals

70b841c

deopt when unpacking aliases and union contains alias

cf2259d

MichaReiser force-pushed the micha/union-builder-sub-unions branch from 7b6d0a9 to cf2259d Compare December 25, 2025 15:56

Try claude suggested fix

54347a5

MichaReiser force-pushed the micha/union-builder-sub-unions branch from 14dc667 to 54347a5 Compare December 27, 2025 13:54

MichaReiser mentioned this pull request Jan 3, 2026

[ty] Avoid UnionBuilder overhead when creating a new union from the filtered elements of an existing union #22352

Merged

MichaReiser closed this Jan 21, 2026

MichaReiser mentioned this pull request Jan 21, 2026

Skip redundancy checks when adding a union to another union astral-sh/ty#2577

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ty] Skip redundancy checks for sub-union-elements#22071

[ty] Skip redundancy checks for sub-union-elements#22071
MichaReiser wants to merge 4 commits intomainfrom
micha/union-builder-sub-unions

MichaReiser commented Dec 19, 2025 •

edited

Loading

Uh oh!

astral-sh-bot bot commented Dec 19, 2025 •

edited

Loading

Uh oh!

astral-sh-bot bot commented Dec 19, 2025 •

edited

Loading

Uh oh!

codspeed-hq bot commented Dec 19, 2025 •

edited

Loading

Uh oh!

MichaReiser commented Dec 19, 2025

Uh oh!

MichaReiser commented Dec 19, 2025

Uh oh!

AlexWaygood commented Dec 19, 2025

Uh oh!

MichaReiser commented Dec 19, 2025

Uh oh!

MichaReiser commented Jan 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

MichaReiser commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

astral-sh-bot bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Diagnostic diff on typing conformance tests

Uh oh!

astral-sh-bot bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mypy_primer results

Uh oh!

codspeed-hq bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #22071 will improve performance by 32.09%

Summary

Benchmarks breakdown

Footnotes

Uh oh!

MichaReiser commented Dec 19, 2025

Uh oh!

MichaReiser commented Dec 19, 2025

Uh oh!

AlexWaygood commented Dec 19, 2025

Uh oh!

MichaReiser commented Dec 19, 2025

Uh oh!

MichaReiser commented Jan 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MichaReiser commented Dec 19, 2025 •

edited

Loading

astral-sh-bot bot commented Dec 19, 2025 •

edited

Loading

astral-sh-bot bot commented Dec 19, 2025 •

edited

Loading

`mypy_primer` results

codspeed-hq bot commented Dec 19, 2025 •

edited

Loading