[ty] Infer the Python version from `--python=<system installation>` on Unix #18550

AlexWaygood · 2025-06-08T13:29:27Z

Summary

If a user passes --python=<virtual_environment> and doesn't specify a Python version on the CLI or in a configuration file, we currently use the metadata in that virtual environment's pyvenv.cfg file to infer the Python version we should use for resolving modules and types. We currently do not attempt to infer the Python version if a user passes --python=<system_installation>, but it's easy enough to do that as well, providing the user is not on a Windows machine. This PR implements that.

Test Plan

CLI integration tests, and manual testing.

Screenshot of what this looks like on the CLI:

github-actions · 2025-06-08T13:34:26Z

`mypy_primer` results

No ecosystem changes detected ✅

github-actions · 2025-06-08T13:39:45Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

MichaReiser

What makes it hard if the user is on a windows machine? I'd prefer if the CLI has consistent behavior across platforms.

AlexWaygood · 2025-06-08T14:04:33Z

What makes it hard if the user is on a windows machine?

Apologies — I left several comments about this in the source code but forgot to mention it in my PR description.

On Unix, the site-packages directory is always located at <sys.prefix>/lib/pythonX.Y/site-packages (with minor variations if it's a PyPy installation or a free-threaded installation), so we can just look at the parent directory of the resolved site-packages directory to get the information we need. On Windows, the site-packages directory is always located at <sys.prefix>/Lib/site-packages, so we can't infer the information we need from the directory structure of the installation.

I'd prefer if the CLI has consistent behavior across platforms.

Yeah, I also wasn't totally sure if this was worth the inconsistency. I think being able to infer the Python version correctly from the minimum number of config options is very high-value, though, and chatting with @zanieb at PyCon they also thought that this was probably worth it

zanieb · 2025-06-09T14:59:36Z

Yeah on Windows you'd need to query the interpreter to infer the version. Or.. maybe you could get away with reading the registry and matching paths (but that seems more complicated than it's worth here!)

AlexWaygood · 2025-06-09T15:05:10Z

Or.. maybe you could get away with reading the registry and matching paths (but that seems more complicated than it's worth here!)

yeah I definitely don't want to do that 😆

MichaReiser

Ty

Mainly a few comments around terminology.

MichaReiser · 2025-06-09T16:11:45Z

crates/ruff_python_ast/src/python_version.rs

+        let (major, minor) = s.split_once('.').ok_or(())?;
+        Self::try_from((major, minor)).map_err(|_| ())


Nit: Not sure if it's worth it because it's so simple but this is somewhat duplicated now with the serde deserialization logic below.

This was bothering me too, especially because we also had duplicated logic in this third place:

ruff/crates/ty_python_semantic/src/module_resolver/typeshed.rs

Lines 331 to 346 in 3aae1cd

fn python_version_from_versions_file_string(

s: &str,

) -> Result<PythonVersion, TypeshedVersionsParseErrorKind> {

let mut parts = s.split('.').map(str::trim);

let (Some(major), Some(minor), None) = (parts.next(), parts.next(), parts.next()) else {

return Err(TypeshedVersionsParseErrorKind::UnexpectedNumberOfPeriods(

s.to_string(),

));

};

PythonVersion::try_from((major, minor)).map_err(|int_parse_error| {

TypeshedVersionsParseErrorKind::IntegerParsingFailure {

version: s.to_string(),

err: int_parse_error,

}

})

}

I've unified the three implementations.

crates/ty/src/args.rs

crates/ty/tests/cli/python_environment.rs

crates/ty_python_semantic/src/module_resolver/resolver.rs

crates/ty_python_semantic/src/program.rs

AlexWaygood · 2025-06-09T16:51:32Z

Apologies for the terrible image quality of this screenshot (taken on an ancient Windows machine I have hanging around that does not have -- and will never have -- Rust installed on it). But it does look like it is possible to query the metadata of a Python binary on Windows without actually invoking that binary in a subprocess. The "File version" and "Product version" fields here both contain the Python version in them (obtained here by right-clicking on the .exe file in Windows Explorer, clicking on "properties", and then nagivating to the "Details" tab).

The metadata appears consistent regardless of whether the binary was installed using "official" python.org installers or using uv.

From what I can tell, though, querying these .exe-file properties from Rust might not be trivial. So I'll definitely pass on it for now.

charliermarsh · 2025-06-09T16:55:03Z

Oh that's cool.

zanieb · 2025-06-09T16:57:25Z

That is cool indeed. cc @konstin we could use this to filter Python binaries more aggressively on Windows without querying the interpreter!

AlexWaygood · 2025-06-09T17:20:28Z

Here's a somewhat higher-quality screenshot since that last one appears to have generated some excitement 😆

…n Unix

MichaReiser · 2025-06-11T11:41:19Z

Is there anything specific you want me to re-review?

AlexWaygood · 2025-06-11T11:45:04Z

Is there anything specific you want me to re-review?

Are you okay with my response in #18550 (comment)?

crates/ruff_python_ast/src/python_version.rs and crates/ty_python_semantic/src/module_resolver/typeshed.rs have also seen significant changes since you last reviewed, as a result of your comment at #18550 (comment)!

MichaReiser · 2025-06-11T15:14:29Z

Sorry, I was in meetings. Do you still want me to re-review or are you okay, considering that you merged the change 😆

AlexWaygood · 2025-06-11T15:18:08Z

Ohh sorry! Still might be good for you to give the changes in crates/ruff_python_ast/src/python_version.rs and crates/ty_python_semantic/src/module_resolver/typeshed.rs a quick once-over (unifying our various PythonVersion deserialization methods) -- happy to revert/modify any of those changes if you dislike them!

MichaReiser · 2025-06-11T16:00:25Z

Okay, I'll do so tomorrow

* main: [ty] Add some "inside string" tests for `object.<CURSOR>` completions [ty] Pull types on synthesized Python files created by mdtest (#18539) Update Rust crate anstyle to v1.0.11 (#18583) [`pyupgrade`] Fix `super(__class__, self)` detection in UP008 (super-call-with-parameters) (#18478) [ty] Generate the top and bottom materialization of a type (#18594) `SourceOrderVisitor` should visit the `Identifier` part of the `PatternKeyword` node (#18635) Update salsa (#18636) [ty] Update mypy_primer doc (#18638) [ty] Improve support for `object.<CURSOR>` completions [ty] Add `CoveringNode::find_last` [ty] Refactor covering node representation [ty] Infer the Python version from `--python=<system installation>` on Unix (#18550) [`flake8-return`] Fix `RET504` autofix generating a syntax error (#18428) Fix incorrect salsa `return_ref` attribute (#18605) Move corpus tests to `ty_python_semantic` (#18609) [`pyupgrade`] Don't offer fix for `Optional[None]` in non-pep604-annotation-optional (`UP045)` or non-pep604-annotation-union (`UP007`) (#18545) [`pep8-naming`] Suppress fix for `N804` and `N805` if the recommend name is already used (#18472) [`ruff`] skip fix for `RUF059` if dummy name is already bound (unused-unpacked-variable) (#18509)

AlexWaygood added the ty Multi-file analysis & type inference label Jun 8, 2025

AlexWaygood requested review from MichaReiser, carljm, dcreager and sharkdp as code owners June 8, 2025 13:29

AlexWaygood force-pushed the alex/system-python-version branch from 7a440de to 14ea91b Compare June 8, 2025 13:30

AlexWaygood requested review from zanieb and removed request for carljm, dcreager and sharkdp June 8, 2025 13:31

MichaReiser reviewed Jun 8, 2025

View reviewed changes

MichaReiser approved these changes Jun 9, 2025

View reviewed changes

zanieb mentioned this pull request Jun 9, 2025

Filter Python executables on Windows by Python version metadata astral-sh/uv#13927

Open

AlexWaygood added 8 commits June 11, 2025 10:42

[ty] Infer the Python version from --python=<system installation> o…

112c1e8

…n Unix

consolidate deserialization methods for PythonVersion

7c19203

improve documentation of the test

6c6dc1b

rename field

04bbe62

rename method

b66faa7

remove values from Priority enum

99b9130

call try_resolve_installation_python_version more lazily

cd80351

improve docs for config options

a453efc

fix failing tests

a572b1e

AlexWaygood force-pushed the alex/system-python-version branch from 14ea91b to a572b1e Compare June 11, 2025 10:59

AlexWaygood requested a review from MichaReiser June 11, 2025 11:15

Add debug assertion and comment

9199382

AlexWaygood enabled auto-merge (squash) June 11, 2025 14:29

AlexWaygood merged commit e84406d into main Jun 11, 2025
31 checks passed

AlexWaygood deleted the alex/system-python-version branch June 11, 2025 14:32

		let (major, minor) = s.split_once('.').ok_or(())?;
		Self::try_from((major, minor)).map_err(\|_\| ())

	fn python_version_from_versions_file_string(
	s: &str,
	) -> Result<PythonVersion, TypeshedVersionsParseErrorKind> {
	let mut parts = s.split('.').map(str::trim);
	let (Some(major), Some(minor), None) = (parts.next(), parts.next(), parts.next()) else {
	return Err(TypeshedVersionsParseErrorKind::UnexpectedNumberOfPeriods(
	s.to_string(),
	));
	};
	PythonVersion::try_from((major, minor)).map_err(\|int_parse_error\| {
	TypeshedVersionsParseErrorKind::IntegerParsingFailure {
	version: s.to_string(),
	err: int_parse_error,
	}
	})
	}

[ty] Infer the Python version from --python=<system installation> on Unix #18550

[ty] Infer the Python version from --python=<system installation> on Unix #18550

Uh oh!

Conversation

AlexWaygood commented Jun 8, 2025

Summary

Test Plan

Uh oh!

github-actions bot commented Jun 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mypy_primer results

Uh oh!

github-actions bot commented Jun 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ruff-ecosystem results

Linter (stable)

Linter (preview)

Formatter (stable)

Formatter (preview)

Uh oh!

MichaReiser left a comment

Choose a reason for hiding this comment

Uh oh!

AlexWaygood commented Jun 8, 2025

Uh oh!

zanieb commented Jun 9, 2025

Uh oh!

AlexWaygood commented Jun 9, 2025

Uh oh!

MichaReiser left a comment

Choose a reason for hiding this comment

Uh oh!

MichaReiser Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

AlexWaygood Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexWaygood commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

charliermarsh commented Jun 9, 2025

Uh oh!

zanieb commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlexWaygood commented Jun 9, 2025

Uh oh!

MichaReiser commented Jun 11, 2025

Uh oh!

AlexWaygood commented Jun 11, 2025

Uh oh!

Uh oh!

MichaReiser commented Jun 11, 2025

Uh oh!

AlexWaygood commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MichaReiser commented Jun 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[ty] Infer the Python version from `--python=<system installation>` on Unix #18550

[ty] Infer the Python version from `--python=<system installation>` on Unix #18550

github-actions bot commented Jun 8, 2025 •

edited

Loading

`mypy_primer` results

github-actions bot commented Jun 8, 2025 •

edited

Loading

`ruff-ecosystem` results

AlexWaygood commented Jun 9, 2025 •

edited

Loading

zanieb commented Jun 9, 2025 •

edited

Loading

AlexWaygood commented Jun 11, 2025 •

edited

Loading