venv activate.bat is UTF-8 encoded but uses current console codepage

[Jac0]

BPO	32409
Nosy	@pfmoore, @vsajip, @vstinner, @tjguk, @ezio-melotti, @zware, @eryksun, @zooba, @pablogsal, @miss-islington, @samstagern
PRs	bpo-32409: Ensures activate.bat can handle Unicode contents #5757 [3.7] bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5757) #5765 [3.7] bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5757) #5765 [3.6] bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5757) #5766 bpo-32409 Ensures activate.bat can handle Unicode contents #10295 [3.7] bpo-32409: Fix regression in activate.bat on international Windows (GH-10295) #10377 bpo-32409: Revert "bpo-32409: Fix regression in activate.bat on international Windows (GH-10295)" #10403 [3.7] Revert "bpo-32409: Fix regression in activate.bat on international Windows (GH-10295)" (GH-10403) #10405 bpo-34144: Fix of venv acvtivate.bat for win 10 #8321

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2019-07-26.16:39:28.243>
created_at = <Date 2017-12-22.10:36:15.658>
labels = ['type-bug', '3.8', 'OS-windows', '3.7', 'library', 'expert-unicode']
title = 'venv activate.bat is UTF-8 encoded but uses current console codepage'
updated_at = <Date 2019-07-26.16:39:28.243>
user = 'https://bugs.python.org/Jac0'

bugs.python.org fields:

activity = <Date 2019-07-26.16:39:28.243>
actor = 'steve.dower'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2019-07-26.16:39:28.243>
closer = 'steve.dower'
components = ['Library (Lib)', 'Unicode', 'Windows']
creation = <Date 2017-12-22.10:36:15.658>
creator = 'Jac0'
dependencies = []
files = []
hgrepos = []
issue_num = 32409
keywords = ['patch']
message_count = 18.0
messages = ['308931', '308934', '312356', '312389', '312390', '312393', '329141', '329142', '329425', '329431', '329440', '329447', '329450', '329451', '329464', '329481', '332324', '348505']
nosy_count = 12.0
nosy_names = ['paul.moore', 'vinay.sajip', 'vstinner', 'tim.golden', 'ezio.melotti', 'zach.ware', 'eryksun', 'steve.dower', 'pablogsal', 'Jac0', 'miss-islington', 'Martin Bijl-Schwab']
pr_nums = ['5757', '5765', '5765', '5766', '10295', '10377', '10403', '10405', '8321']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32409'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[Jac0]

Let's say I have a folder c:\test-ä in Windows

Now if I run: py -m venv env
and activate: env\scripts\activate
and check: where python

the result is incorrectly just: C:\Users\Username\AppData\Local\Programs\Python\Python36\python.exe

If I run: path
the result is: PATH=C:\test-ä\env\Scripts;...

So clearly the encoding is broken for the folder name.

I can fix this by changing activate.bat character encoding to OEM-US and then replacing "test-├ż" by "test-ä".

If I now activate and run: where python
the result is (as should be):
C:\test-ä\env\Scripts\python.exe
C:\Users\Username\AppData\Local\Programs\Python\Python36\python.exe

By running: path
I get: PATH=C:\test-ä\env\Scripts;...

So looks good here as well.

I suggest that what ever is creating activate.bat file, is using incorrect character encoding for the creation of the file. If this is somehow platform specific, there could be a guide in the venv documentation about how to fix this.

[eryksun]

The CMD shell decodes batch scripts using the attached console's output codepage, which defaults to OEM. OTOH, venv writes the replacement values for the template activate.bat as UTF-8 (codepage 65001), which is correct and should not be downgraded to OEM.

Instead, the batch script could temporarily switch the console to codepage 65001. Then restore the previous codepage at the end. For example:

@echo off
for /f "tokens=2 delims=:" %%a in ('"%SystemRoot%\System32\chcp.com"') do (
    set "CODEPAGE=%%a"
)
"%SystemRoot%\System32\chcp.com" 65001 > nul

[rest of script]

"%SystemRoot%\System32\chcp.com" %CODEPAGE% > nul
set "CODEPAGE="
:END

[zooba]

Eryk's solution seems to be best, so I'll add that.

[zooba]

New changeset 6240917 by Steve Dower in branch 'master':
bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5757)
6240917

[zooba]

New changeset a3d6c1b by Steve Dower (Miss Islington (bot)) in branch '3.7':
bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5765)
a3d6c1b

[zooba]

New changeset 8e149ff by Steve Dower (Miss Islington (bot)) in branch '3.6':
bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5766)
8e149ff

[samstagern]

it does not work as expected on swiss german (and likely other internationalised) windows systems. See https://bugs.python.org/issue32409

[samstagern]

I meant https://bugs.python.org/issue35148

[vsajip]

New changeset c64583b by Vinay Sajip (samstagern) in branch 'master':
bpo-32409: Fix regression in activate.bat on international Windows (GH-10295)
c64583b

[vsajip]

New changeset 881e273 by Vinay Sajip (Miss Islington (bot)) in branch '3.7':
bpo-32409: Fix regression in activate.bat on international Windows (GH-10295) (GH-10377)
881e273

[pablogsal]

This commit broke a lot of the Windows buildbots:

https://buildbot.python.org/all/#/builders/32/builds/1707
https://buildbot.python.org/all/#/builders/113/builds/733
https://buildbot.python.org/all/#/builders/40/builds/1137
https://buildbot.python.org/all/#/builders/130/builds/420
https://buildbot.python.org/all/#/builders/3/builds/1634

Can someone work on a patch soon? If not we need to revert the commits in all the branches that are failed to build :(

[samstagern]

I will have a look.

[pablogsal]

New changeset 6843ffe by Pablo Galindo in branch 'master':
Revert "bpo-32409: Fix regression in activate.bat on international Windows (GH-10295)" (GH-10403)
6843ffe

[miss-islington]

New changeset 3ba5e25 by Miss Islington (bot) in branch '3.7':
Revert "bpo-32409: Fix regression in activate.bat on international Windows (GH-10295)" (GH-10403)
3ba5e25

[vstinner]

Pablo Galindo reverted the change because it broke Windows buildbots, and we have a policy to revert a change breaking buildbots if the regression cannot be fixed "quickly" event:
https://pythondev.readthedocs.io/ci.html#revert-on-fail

Is someone working on investigating the bug? Do you need help to reproduce the bug?

Copy of the test_venv error:

======================================================================
ERROR: test_unicode_in_batch_file (test.test_venv.BasicTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows8\build\lib\test\test_venv.py", line 302, in test_unicode_in_batch_file
    out, err = check_output(
  File "D:\buildarea\3.x.bolen-windows8\build\lib\test\test_venv.py", line 37, in check_output
    raise subprocess.CalledProcessError(
TypeError: __init__() takes from 3 to 5 positional arguments but 6 were given

https://buildbot.python.org/all/#/builders/32/builds/1707

[zooba]

That error is a bug in the test, but it only shows up on an error path anyway. Without removing the extra None we don't get to see the actual error output.

I can't look into this over the next week or two, but a quick glance at the original PR looks like a lot of quotes are missing around executable paths, so maybe it was that?

[vsajip]

See also bpo-35558.

[zooba]

This appears to be completely resolved now, but if it's not, please ping with the details and we'll reopen.