py ./x.py no longer works on Windows

[CAD97]

Workaround for Windows developers

Put the following as py.ini in %LOCALAPPDATA%:

[commands]
bash=python

This will make py interpret the #!/usr/bin/env bash shebang as running python.

You can also put py.ini alongside py.exe. This is likely C:\Windows; you can find it with where py_cmd or Get-Command py_pwsh.

Alternatively, run x.py with py -3 to explicitly select to use the default Python 3 version rather than looking at the shebang. You should also be able to set ftype Python.File="C:\Windows\py.exe" "-3" "%L" %* so ./x.py will use py -3 to run.

To debug, set $env:PYLAUNCH_DEBUG=1_pwsh/set PYLAUNCH_DEBUG=1_cmd.

---

Issue description

#98474 added a #!/usr/bin/env bash shebang line to x.py to allow ./x.py to work on unixy platforms with python3 but not python (the previous shebang line was #!/usr/bin/env python).

This means that x.py no longer works on Windows machines which run python via the py Python Launcher for Windows, as py tries to interpret the shebang line (in order to use the correct python version) and ends up instead trying to run /usr/bin/env, which obviously doesn't exist on a Windows machine. Explicitly asking for a specific version from py (e.g. by running py -3 for the latest 3.x version) will successfully run the script, as then py ignores the shebang line.

In theory this is an upstream bug; py should not be trying to run something that isn't python. In practice, though, this is how most Windows users will be launching python, as this is the launcher set up to handle .py files when installing python from the official installer from the website or Python.Python.3 in winget. (The msstore python which the default-installed python and python3 shims point to reportedly do not install the py launcher.) Additionally, this worked fine previously, and only broke with #98474.

#71818 (comment) reports that #!/usr/bin/env python didn't work on Windows; I believe this is due to the order of events used in the described testing. @Walther first installed the msstore python. They then uninstalled the msstore python and installed the official python distribution, with "add python 3.10 to PATH" unchecked, reporting it to be unchecked by default. However, it was my experience that the official distribution did add python to the path by default; I suspect that the reason it didn't work for Walther was because (parts of) the msstore python was still in PATH, and the installer did not want to overwrite an existing installation. With my installation, a python and python3 shebang both work, and a python2 shebang results in a "Requested Python version (2) is not installed".

Changing the shebang to python3 has previously been delayed due to concerns about Windows compatibility. The specific situation seems to be this:

• When python is handled through the py.exe launcher (i.e. the shell uses the ftype registered handler), all python virtual commands will seamlessly work on Windows.
• However, when using a MSYS-like shell (e.g. Git Bash) which directly interprets shebangs, it will try to run the python3 program, which most likely does not exist on a Windows machine. (E.g. in my winget install of Python.Python.3, I have python.exe and python3.dll in path, but no python3.exe.)

[Rageking8]

@rustbot label C-bug

[CAD97]

Since this is also relevant, from the Using Python on Windows docs:

There are a number of different installers available for Windows, each with certain benefits and downsides.

The full installer contains all components and is the best option for developers using Python for any kind of project.

The Microsoft Store package is a simple installation of Python that is suitable for running scripts and packages, and using IDLE or other development environments. It requires Windows 10, but can be safely installed without corrupting other programs. It also provides many convenient commands for launching Python and its tools.

[...]

The Microsoft Store package is an easily installable Python interpreter that is intended mainly for interactive use, for example, by students.

As much as MS would like to push the msstore version of python, the full installer is still generally preferred for developers.

(Also, thanks, I used the blank template because I wasn't sure what to use for build tooling.)

[Rageking8]

@rustbot label A-rustbuild

[dtolnay]

While this is being shaken out in python/cpython#94399 (I agree with python/cpython#94399 (comment) — ideally this would be at most a warning, and py should interpret the file as if there were no shebang, which I believe already works):

Is there a different possible entrypoint with better availability across systems? python/cpython#94399 (comment) explains that py would look for it inside of "%SystemRoot%\System32" or "%SystemRoot%\SysWOW64". Perhaps perl? @CAD97 do you notice anything promising-looking in those directories that might exist on most platforms in a default installation?

[dtolnay]

From the discussion on the cpython issue it sounds like this is already fixed in the upcoming Python 3.11's py launcher. The release is scheduled for October 2022.

We can decide whether to revert our x.py change until October. I am hesitant because it would involve re-breaking the Linux and macOS users, while Linux distributions (since 2020) and macOS upgrades (since October 2021) are deleting python.

[CAD97]

§ 4.8.4.1 Customization via INI files provides a workaround: py.ini can be used to specify that a bash shebang should instead run {insert executable}. Unfortunately for us at least, py.ini is loaded from %LOCALAPPDATA% and the same directory py.exe is in (e.g. C:\Windows with a default install), so we can't just drop a py.ini next to x.py to fix the behavior of py.

[commands]
bash=python

From the discussion on the cpython issue it sounds like this is already fixed in the upcoming Python 3.11's py launcher.

I'm not 100% sure whether this is considered a feature or a bug in the new launcher at the moment, though, given the comment about not changing the old launcher's behavior, even though they said the no-env-lookup behavior might make sense in the new launcher...

We can decide whether to revert our x.py change until October.

A different alternative would be to change the shebang to python3 in the interim; this was previously rejected as it was thought to break Windows, but in theory it shouldn't¹. This would instead break old unixy users who were partially broken by only having python2 available, as well as MSYS users who do not have a python3 available (but imho if you're running a unixy environment, you should probably have python3, or other things will probably cause issues as well.)

do you notice anything promising-looking in those directories that might exist on most platforms in a default installation?

Unless cmd exists on unixy systems... probably not. (My system is fairly clean; I did a clean install of Win11 around the start of summer.)

A list of maybe-potentially-useful-but-probably-not executables I spotted scrolling through:

Yes this list includes pretty obviously non-useful commands. I picked basically anything with even a vaguely not-windows-specific name in System32.

• cmd
• curl
• help
• ipconfig
• mode
• more
• notepad
• nslookup
• perfmon
• ping
• print
• rdpshell
• rdpsign
• replace
• reset
• route
• runas
• runonce
• shutdown 🙃
• sort
• timeout
• where
• whoami
• wsl (note: installer. I don't have WSL installed on this machine yet, otherwise I'd probably have a bash available)
• xcopy

Footnotes

1. As I uninstalled the python3 shim to msstore, I can't test to see if py would detect and try to launch that instead of an actual python interpreter. py will attempt to use msstore-installed python if the full installer python is not available, so it's possible the shim would be selected. ↩

[wesleywiser]

I'm probably missing something obvious (I have basically 0 Python experience), but why can't we just document that Windows users do python x.py ...? That works both before and after this change.

[ChrisDenton]

If you use the python.org installer with the default options then it won't add python to the path. That said, it is just a tick box to select it.

[ChrisDenton]

Btw, I feel like I mentioned the failure mode in the OP on the original PR but perhaps I wasn't clear enough.

[jyn514]

I'm probably missing something obvious (I have basically 0 Python experience), but why can't we just document that Windows users do python x.py ...? That works both before and after this change.

This change broke existing workflows. Existing contributors aren't looking at documentation, and it seems weird to put the responsibility on them.

[dtolnay]

Existing contributors aren't looking at documentation, and it seems weird to put the responsibility on them.

I think this is an oversimplification; we regularly put responsibility on existing contributors. If an existing contributor's muscle-memory workflow breaks, most of them will not quit the project over it. They will look somewhere for guidance. The next step in this issue is figuring out where they look for guidance (probably directly in x.py, first), and putting guidance there.

[wesleywiser]

I'm probably missing something obvious (I have basically 0 Python experience), but why can't we just document that Windows users do python x.py ...? That works both before and after this change.

This change broke existing workflows. Existing contributors aren't looking at documentation, and it seems weird to put the responsibility on them.

In general, I agree with this point and I'm also frustrated when my workflows are interrupted so I definitely empathize. We should absolutely strive to minimize workflow changes to project contributors!

At the same time, it should probably be noted that at no point (from what I can see) have we ever told people to run py x.py on Windows-msvc (I have no knowledge of Windows-gnu and never use it), at least in the Readme.md. The file says to run python x.py ... and that line blames back to the introduction of the Python based build system 6 years ago.

Overall it's becoming increasingly clear to me that we need to move away from the current Python based system because of problems like this. @jyn514 had a fantastic idea there and I'm very excited to see that happen! Until that time though, we're very much stuck between a rock and a hard place. The reasons put forth in #98474 still seem compelling to me so I would prefer (speaking only for myself, personally) that we try to find a way to make the error message clearer as @jyn514 mentioned rather than reverting.

[CAD97]

Here's I think a near-full summary of the Windows behavior as it currently exists:

Before (#!/usr/bin/env python)

• ./x.py (pwsh; x.py in cmd) works iff an ftype handler has been registered for .py; this is the case if the “full installer” was used, or if the msstore python was used and a default “Open With” has been manually set to a python interpreter.
• With a “full installer” python, py ./x.py and python ./x.py work; python2 ./x.py and python3 ./x.py do not.
• With msstore python, python2 or python3 (as applicable) may work; I tested installing msstore python and did not get a python3 exe in path, but this behavior may be different if the default python3 shim which launches msstore has not been uninstalled.

After (#!/usr/bin/env bash)

• py ./x.py (and ./x.py if py is the ftype registration) attempt to emulate the behavior of env bash, failing with an unhelpful error if bash is not in path, trying to run /usr/bin/env. (If a bash is available, it is run.)
• python ./x.py, python3, and python2 work as before.

Which shebangs work on Windows?

Through py.exe, any /usr/bin/env python{version} is documented as launching via py -{version}; this applies with any vcommand shebang. Through an MSYS like shell, the shebang is always interpreted and the env program is always run like on any other POSIX shell.

As such, #! /usr/bin/env python is the most Windows-portable shebang line. python3 is probably also completely acceptable; imho it is not unreasonable to expect a POSIXy shell environment on Windows to provide python3 as a (soft) alias to (the logic of) py -3.

#! /usr/bin/env x where env x fails lookup may start to call the default python when launched via py.exe with Python 3.11+, and may even skip path lookup entirely. It's impossible to say for sure until it releases whether the current implemented behavior diverging from the existing launcher is desired or will be changed to match it. However, trying to envoke /usr/bin/env does seem to be considered a bug.

[CAD97]

If an existing contributor's muscle-memory workflow breaks, most of them will not quit the project over it. They will look somewhere for guidance. The next step in this issue is figuring out where they look for guidance (probably directly in x.py, first), and putting guidance there.

Actually, I first blamed nushell 🙃 It was only after someone else prompted me that I checked in pwsh and realized the error was not coming from the shell and then went to check x.py and spotted the changed shebang.

However, even though the contributing guidelines say to run python ./x.py, using py is generally understood as the proper way to launch python on Windows¹, the same way a unix user might translate documentation suggesting using python into python2 or python3.

At least to this Windows user, it seems the “more correct” behavior for x.py is to ask for python as “some python” and detect and handle getting python2 instead of python3, or to up front ask for python3, than it is to ask for bash and polyglot-relaunch with python3. I understand the practicality of distros removing python and requiring an explicit python-is-pythonN install, but losing the ability to ask for “python2 or python3, I'll handle it” is unfortunate.

Then ignoring the bootstrap problem entirely, it'd be nice for at least some directories to “just work” under the system/rustup cargo. If e.g. x.py is required for --stage 1, but rustup-cargo “just works” for --stage 0 development/testing², then x.py and any weirdness about getting it to run³ becomes less important for most uses.

Footnotes

1. If it weren't intended to be used like this, but only as an ftype handler to enable ./x.py to work like on POSIX platforms, then it shouldn't have been added to PATH. ↩

2. e.g. by a rustup toolchain toml which asks for the bootstrap compiler to be selected through rustup and sets an undocumented i_am_rustc_bootstrap key to enable the RUSTC_BOOTSTRAP=1 behavior ↩

3. I don't know what @jyn514's plan is, but a cargo-xtask style alias for cargo x to cargo run the installable version of the x script would also eliminate the need to use python/x.py entirely, unless doing a bootstrap build without a preinstalled toolchain. That is still a thing which should be supported, but the less common it is, the greater the allowance we have for requiring slightly nonstandard workflows. ↩

[m-ou-se]

Wouldn't it be easier to just add a separate x.bat for Windows? (Or whatever other file extension and script language works out of the box on Windows.)

[ChrisDenton]

I believe the possibility of x.bat/x.ps1 (on Windows) and x.sh (on POSIX) has been discussed a few time before, though search is failing me atm.

[CAD97]

In one of the other threads I commented on x.bat (there x.ps1) to the extent of roughly:

To this Windows user, it seems to make more sense to have an x.sh than an x.bat. x.py with a python3 shebang works for (somewhat less than) everyone, since it's just a regular python file using the python interpreter as its interpreter.

The actual broken workflows with a python3 shebang that maybe would've worked before is just any distro with python defined but not python3. Two reminders on that: it will still work if those users use python ./x.py, and py.exe transparently does the right thing with any python{...} shebang.

But why not make ./x.py work for ~all POSIX systems (iow the ones that would normally expect ./script-file to work) via the (ba)sh polyglot? Well, if you're providing shell-specific entrypoints anyway (e.g. x.bat), imho it makes sense then to have the sh-specific trampoline in x.sh instead of polyglotted at the head of x.py.

Maybe that logic doesn't hold, since "anyone" respecting the shebang "should" have (ba)sh available. (Caveat: win32 + py.exe exists, thus the discussion at all.) Maybe the greater number of x.py users on unixy platforms than Windows means papering over the python split for unixy users is more worthwhile. But it's also worth noting in this comparison the failure modes as well: for Windows + #!/usr/bin/sh it's Unable to create process using '/usr/bin/env bash ./x.py'; for unixy + #!/usr/bin/env python3 it's /usr/bin/env: ‘python3’: No such file or directory (and perhaps some distros will even have a hint to apt install python3).

The failure mode of #!/usr/bin/python3 on unixy platforms is clear, and likely an error message that a unix user already knows how to interpret the meaning of. The failure mode of #!/usr/bin/sh on Windows is opaque, and something very few users will have any¹ intuition for what went wrong.

IMHO, this makes it fairly clear that using a #!/usr/bin/python3 shebang (and providing an x.sh trampoline) is superior to using a #!/usr/bin/sh (and providing an x.bat trampoline). But as someone on Windows, I'm obviously biased. But also, I've already set up py.ini to pretend bash is python, so the choice here has little impact on me; it's more backed by trying to minimize the confusion (and importantly, new developer bounce rate²) of the failure case messaging.

Footnotes

1. As a narcissistic example, I'm a reasonably techy Windows user with experience using POSIX systems and tooling. My first reaction was to blame something going wrong in my (admittedly, nonstandard) shell rather than consider that it was somehow the script's fault. ↩

2. If anything, I'm additionally worried that this will give Windows users trying to first-time hack on the compiler a "oh, this only builds on unix" impression if they see [...] /usr/bin/sh [...] in an error message, and just leave before looking to see if there's another way to build³. It's not an uncommon thing to deal with for Windows devs that even portable software only really builds properly on unix. Whereas, for better or for worse, linux-on-the-desktop developers are much more likely to see their error message and have the takeaway of "the build system requires python3." ↩

3. Again, it's not uncommon for when this exists for it to be a poorly maintained blob of Visual Studio build instructions (if you're lucky, an automated translation from the actual build system) which work less well than the real unix build system. ↩

[Kobata]

But it's also worth noting in this comparison the failure modes as well: for Windows + #!/usr/bin/sh it's Unable to create process using '/usr/bin/env bash ./x.py'

It's worth pointing out here that on Windows with WSL the 'failure mode' for #!/usr/bin/env bash is instead that... it runs, except under WSL (py.exe handles #!/usr/bin/env so it finds the %LOCALAPPDATA%\Microsoft\WindowsApps\bash.exe launcher, and runs that) which leads to much fun of things mostly working except for sometimes git submodule updates failing and the platform detection defaulting to linux (because, it is running under linux)

[ChrisDenton]

That seems like a serious failure mode. I hadn't considered that.

(Aside: I find it a bit weird that the python launcher will launch something that isn't python. But it is what it is.)

[CAD97]

(Aside: I find it a bit weird that the python launcher will launch something that isn't python. But it is what it is.)

Being explicit: the behavior is now tracked via a GitHub issue (python/cpython#94399). The seemingly most likely upstream classification is that

• The behavior of py.exe interpreting shebangs with a non-python executable is undocumented behavior.
• Changing the behavior of the existing launcher of searching path when given a /usr/bin/env is likely a nonstarter.
  • It's more than that, even; #! x will use x as the interpreter. Classic py.exe is kinda a shebang interpreter first.
  • For some recursive fun, set a #! py shebang...
• Changing the behavior when the path search fails to output a specific error rather than asking the OS to run /usr/bin/env is reasonable.
• The new launcher (to be provided with Python 3.11+) can (and currently does) remove the undocumented path search behavior, just using the default python interpreter.

[scottmcm]

but why can't we just document that Windows users do python x.py ...?

FWIW, as a contributor using Windows, that's what I've always done. I didn't know py x.py was even a thing!

[Stargateur]

same for me, I often work on windows but use cygwin that doesn't have any problem with the change.

[CAD97]

Update on upstream: the new launcher has reimplemented the /usr/bin/env PATH lookup (though in a way that fails much more gracefully when lookup fails) and also considers the shebang in a *.py to always be a python interpreter.

If someone creates a .py file and gives it a shebang to call bash, that person is lying about what their file does :-)

If we want to avoid x.py always running under WSL on Windows when bash launches under WSL, we'll need to do something along the lines of #99992.

[jyn514]

A summary of the current situation: #99992 changed the shebang from #!/usr/bin/env bash to #!/usr/bin/env python3. For systems without python3 installed, there are new x and x.ps1 scripts which will find an appropriate version of python.

If you find that none of these options work for you, or the new situation is more inconvenient for you than before, please comment here or open a new issue :)

[CAD97]

As a procedural note, this should probably be unpinned in perhaps about a week if nobody reports an issue with the fix/mitigation/direction chosen. Ofc a new issue can always be opened after (or even before) that.