Support -3.11-arm64 in py.exe launcher

[zooba]

BPO	46566
Nosy	@brettcannon, @pfmoore, @tjguk, @zware, @zooba, @tommyvct, @conioh
PRs	bpo-46566: Add new py.exe launcher implementation #32062 bpo-46566: Make test_launcher more robust to a variety of installs #32204
Files	launcher.msi

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2022-03-28.23:22:12.800>
created_at = <Date 2022-01-28.16:52:59.083>
labels = ['type-feature', 'OS-windows', '3.11']
title = 'Support -3.11-arm64 in py.exe launcher'
updated_at = <Date 2022-03-30.21:18:48.191>
user = 'https://github.com/zooba'

bugs.python.org fields:

activity = <Date 2022-03-30.21:18:48.191>
actor = 'steve.dower'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2022-03-28.23:22:12.800>
closer = 'steve.dower'
components = ['Windows']
creation = <Date 2022-01-28.16:52:59.083>
creator = 'steve.dower'
dependencies = []
files = ['50701']
hgrepos = []
issue_num = 46566
keywords = ['patch']
message_count = 16.0
messages = ['412008', '412481', '414299', '414928', '414929', '414936', '415165', '415168', '415173', '415178', '415181', '415844', '416009', '416228', '416229', '416399']
nosy_count = 7.0
nosy_names = ['brett.cannon', 'paul.moore', 'tim.golden', 'zach.ware', 'steve.dower', 'TommyVCT', 'conio']
pr_nums = ['32062', '32204']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue46566'
versions = ['Python 3.11']

[zooba]

We now have a traditional installer that runs on Windows ARM64 devices, but it omits the py.exe launcher. This is because sys.winver is "3.11-arm64" and hence the registry tag (see PEP-514) is not detected by the launcher.

Currently, the launcher is not included in the installer and appears as disabled with an explanatory note.

We need to add support for it eventually, and possibly block the final/RC/beta releases if we don't have it. (We don't ship the launcher with Nuget or Store packages, so nobody will miss it there, but they probably will here.)

32-bit apps can still run on ARM64, so it isn't essential to make it a native process. It just has to be able to detect the registry keys.

It should still support 32-bit and 64-bit installs, because those can also be run on ARM64 devices (though there are numerous things that don't work as well as a native build).

Possibly we want to just do the work to directly follow PEP-514 now? Depends on who volunteers to work on this, I guess.

[tommyvct]

PEP-514 looks good to me

[zooba]

I'm working on this now.

[zooba]

I'm working on what's become a rewrite of the launcher. If anyone would like to follow along, you can see my changes at main...zooba:bpo-46566

It's still missing some functionality, and I'm not sure it's any faster or less complex than the previous version :) But it should handle ARM64 properly, and also knows how to do an install of a requested version if you don't already have it (still TBD whether that stays)

[zooba]

Hah, that's funny URL formatting. Let's see if this is any better:

<main...zooba:bpo-46566\>

[zware]

If it's already turning into a rewrite, how feasible would it be to adopt Brett's py launcher?

[zooba]

If it's already turning into a rewrite, how feasible would it be to adopt Brett's py launcher?

I looked at it already, and I'd have to write literally the same code to implement what's needed :) (as well as learning Rust and convincing everyone to let us use Rust in CPython...). Practically nothing can be reused - there's no registry, no shebang handling, and our process launching on Windows is already very complex (and has to remain that way for compatibility).

For now, the old launcher will remain to be used for venv redirectors, but I've got the setup in the new one to be able to play the same role, as well as potentially being able to be a script or .pyz launcher with a simple rename.

If Brett's proposal for extensions (other executables on PATH that know how to identify Python installs) happens, we'll probably copy it, though PEP-514 covers Windows adequately (and the rewrite allows non-PythonCore installs to be found/launched). But the codebase itself isn't helpful.

[pfmoore]

as well as potentially being able to be a script or .pyz launcher with a simple rename.

Would it be possible to also make the launcher work when prepended to a zipfile? That's a really useful use-case (make a zipapp automatically runnable, but still a single file) that at the moment needs a 3rd party launcher (Vinay's simple-launcher project).

If not, then that's fine, but if we're already doing a significant rewrite that might be a good time to add it.

[zooba]

I'd like to, the main challenge with that is it'd invalidate the code signature on the file, which will make it basically unusable (at the very least, you'll get warnings). A simple rename does not.

But yeah, it can probably go in. Hopefully my restructure will make it easier to follow where certain things get checked.

[brettcannon]

"Practically nothing can be reused - there's no registry, no shebang handling, and our process launching on Windows is already very complex (and has to remain that way for compatibility)."

I do process the shebang to restrict searching, or did you mean something else?

And registry support is planned.

[zooba]

I do process the shebang to restrict searching, or did you mean something else?

That's what I meant. Guess I missed seeing it when scanning the code (probably I should've read the docs :D )

And registry support is planned.

It's attached to a milestone with no due date and 0% completion ;) Being aware that you'll need to do it isn't the same as planning to do it.

[zooba]

Posted my WIP as a draft PR. I'm still adding docs, tests and ironing out minor issues, but the core functionality is there (as well as some added bonuses). Feel free to review, and importantly to mark anywhere you'd like more explanatory comments.

Currently the main known issue is that the "-3-32" (and now-deprecated "-3-64") arguments are not supported. But they will be! There are also likely to be a range of shebang templates that need better handling, and I'll go through the open issues to see what ought to be fixed there compared to today's version.

[zooba]

I think this PR is "ready enough" to get in for broader testing, but in case anyone wants to try it first, I've attached a build from my own machine.

You may need Programs & Features to remove any existing "Python Launcher" before installing this MSI, and you will need to do that to remove this later on, but it should otherwise behave identically to a per-user installed launcher.

[zooba]

New changeset bad86a6 by Steve Dower in branch 'main':
bpo-46566: Add new py.exe launcher implementation (GH-32062)
bad86a6

[zooba]

It's in! Any new issues probably won't be found until the next release, so I'll close this and we can open new bugs.

[zooba]

New changeset f3d5715 by Steve Dower in branch 'main':
bpo-46566: Make test_launcher more robust to a variety of installs (GH-32204)
f3d5715