Python init template does not work on Windows

[Doug-AWS]

1. Python 3.7 installs python.exe, not python3.exe

2. There is no 'source' command in Windows

3. The paths shown in the README.md file are Linux/Unix/Mac, not Windows paths

[garnaat]

Does the init command fail? Or does it just not create the virtualenv?

[Doug-AWS]

It fails as is cannot find python3:

C:>mkdir hello-cdk

C:>cd hello-cdk

C:\hello-cdk>cdk init --language python
Applying project template app for python
Initializing a new git repository...
warning: LF will be replaced by CRLF in README.md.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in app.py.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in cdk.json.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in hello-cdk/hello-cdk_stack.py.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in requirements.txt.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in setup.py.
The file will have its original line endings in your working directory
Executing python -m venv .env
'python3' is not recognized as an internal or external command,
operable program or batch file.
python3 -m venv .env failed: python3 exited with status 1

[garnaat]

But does it create the git repo and copy the files over? If so, can you just manually run the virtualenv command and proceed or does it leave things in a funky state?

[Doug-AWS]

I get:

C:\hello-cdk>dir
Volume in drive C is OSDisk
Volume Serial Number is E8B7-B780

Directory of C:\hello-cdk

04/04/2019 12:20 PM

.
04/04/2019 12:20 PM ..
04/04/2019 12:20 PM 156 app.py
10/26/1985 01:15 AM 32 cdk.json
04/04/2019 12:20 PM hello-cdk
10/26/1985 01:15 AM 1,082 README.md
10/26/1985 01:15 AM 5 requirements.txt
04/04/2019 12:20 PM 1,025 setup.py
5 File(s) 2,300 bytes
3 Dir(s) 718,458,163,200 bytes free

C:\hello-cdk>dir hello-cdk
Volume in drive C is OSDisk
Volume Serial Number is E8B7-B780

Directory of C:\hello-cdk\hello-cdk

04/04/2019 12:20 PM

.
04/04/2019 12:20 PM ..
04/04/2019 12:20 PM 212 hello-cdk_stack.py
10/26/1985 01:15 AM 0 init.py
2 File(s) 212 bytes

[Doug-AWS]

and .git

[Doug-AWS]

Okay, trying to get this to work on Windows is more trouble than doing it all by hand.

For now I recommend we follow instructions similar to what we used back in 0.20.0, https://awslabs.github.io/aws-cdk/versions/0.20.0/getting-started.html:

1. Install the cdk
2. Create hello-cdk and cd there
3. Initialize git and npm
4. Add build and watch to package.json
5. Create cdk.json
6. Create requirements.txt and install the packages w/pip
7. Create app.py

[garnaat]

I guess I don't really see how that's easier. You just run the cdk init and either the virtualenv works and you are good to go or it doesn't and you can then choose to run it manually or just not bother.

[Doug-AWS]

I installed cdk v 0.28.0 as the Python init template was not available until 0.28.0

I created hello-cdk directory and cd'd into it

I ran "cdk init --language python" and got the error message discussed previously

I changed "python3" to "python" in cdk.json and app.py

I ran "pip install -r requirements.txt"

I ran "cdk synth" and got an error:

File "app.py", line 5
    from hello-cdk.hello-cdk_stack import PyStack
                   ^
SyntaxError: invalid syntax
Subprocess exited with error 1

1. I noticed app.py had:
   from cdk_test.cdk_test_stack import PyStack

   but there is no PyStack in hello-cdk\hello-cdk\hello-cdk_stack.py. There is a HelloCdkStack, so I changed the import statement to:

   from cdk_test.cdk_test_stack import HelloCdkStack

   and the initialization of the stack from:

    PyStack(app, "hello-cdk-cdk-1")
   

To:

HelloCdkStack(app, "hello-cdk-cdk-1")

And reran "cdk synth". No love. I got a similar error:

File "app.py", line 5
    from hello-cdk.hello-cdk_stack import HelloCdkStack
              ^
SyntaxError: invalid syntax
Subprocess exited with error 1

So I just changed app.py to:

#!/usr/bin/env python

from aws_cdk import cdk

class HelloCdkStack(cdk.Stack):

def __init__(self, app: cdk.App, id: str, **kwargs) -> None:
    super().__init__(app, id)

    # The code that defines your stack goes here

app = cdk.App()
HelloCdkStack(app, "hello-cdk-cdk-1")

app.run()

And re-ran "cdk synth". This time I got:

Traceback (most recent call last):
File "app.py", line 3, in
from aws_cdk import cdk
File "D:\Python\Python37\lib\site-packages\aws_cdk\cdk_init_.py", line 12, in
import aws_cdk.cx_api
File "D:\Python\Python37\lib\site-packages\aws_cdk\cx_api_init_.py", line 11, in
jsii_assembly = jsii.JSIIAssembly.load("@aws-cdk/cx-api", "0.28.0", name, "[email protected]")
File "D:\Python\Python37\lib\site-packages\jsii_runtime.py", line 40, in load
kernel.load(assembly.name, assembly.version, os.fspath(assembly_path))
File "D:\Python\Python37\lib\site-packages\jsii_kernel_init.py", line 184, in load
self.provider.load(LoadRequest(name=name, version=version, tarball=tarball))
File "D:\Python\Python37\lib\site-packages\jsii_kernel\providers\process.py", line 326, in load
return self._process.send(request, LoadResponse)
File "D:\Python\Python37\lib\site-packages\jsii_utils.py", line 24, in wrapped
stored.append(fgetter(self))
File "D:\Python\Python37\lib\site-packages\jsii_kernel\providers\process.py", line 321, in _process
process.start()
File "D:\Python\Python37\lib\site-packages\jsii_kernel\providers\process.py", line 269, in start
env=environ,
File "D:\Python\Python37\lib\subprocess.py", line 775, in init
restore_signals, start_new_session)
File "D:\Python\Python37\lib\subprocess.py", line 1119, in _execute_child
args = list2cmdline(args)
File "D:\Python\Python37\lib\subprocess.py", line 530, in list2cmdline
needquote = (" " in arg) or ("\t" in arg) or not arg
TypeError: argument of type 'WindowsPath' is not iterable
Exception ignored in: <function _NodeProcess.del at 0x000001C51357B488>
Traceback (most recent call last):
File "D:\Python\Python37\lib\site-packages\jsii_kernel\providers\process.py", line 213, in del
self.stop()
File "D:\Python\Python37\lib\site-packages\jsii_kernel\providers\process.py", line 275, in stop
self._process.terminate()
AttributeError: '_NodeProcess' object has no attribute '_process'
Subprocess exited with error 1

Needless to say, I'm quitting now.

[garnaat]

see #2183 and #2180.

Also, you might want to try cdk init --language python sample-app since that will provide something closer to what the other languages provide by default (see also #2188) and doesn't suffer from these issues. It is also probably a better getting started experience.

I'll fire up a Windows machine to test with.

[Doug-AWS]

I tried adding sample-app, but similar results, although at least app.py had the correct reference to the stack class.

[Doug-AWS]

I linked python3.exe to python.exe on my Windows box ("mklink python3.exe python.exe" as admin) and ran "cdk init --language python" and got:

Applying project template app for python
Initializing a new git repository...
warning: LF will be replaced by CRLF in README.md.
The file will have its original line endings in your working directory
...
Executing python -m venv .env
Error: [WinError 2] The system cannot find the file specified
python3 -m venv .env failed: python3 exited with status 1

So I ran "python -m venv .env", then "cdk ls" and got:

File "app.py", line 5
from hello-cdk.hello-cdk_stack import PyStack
^
SyntaxError: invalid syntax
Subprocess exited with error 1

Any ideas?

[garnaat]

Okay, I finally got a Windows EC2 instance all configured to do some testing. Here's what I did.

PS C:\Users\Administrator> mkdir fiebaz
PS C:\Users\Administrator> cd fiebaz
PS C:\Users\Administrator\fiebaz> cdk init --language=python sample-app
Applying project template sample-app for python
Initializing a new git repository...
warning: LF will be replaced by CRLF in README.md.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in app.py.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in cdk.json.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in hello/hello_construct.py.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in hello/hello_stack.py.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in requirements.txt.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in setup.py.
The file will have its original line endings in your working directory
warning: LF will be replaced by CRLF in tests/unit/test_hello_construct.py.
The file will have its original line endings in your working directory
Executing python -m venv .env
'python3' is not recognized as an internal or external command,
operable program or batch file.
python3 -m venv .env failed: python3 exited with status 1

PS C:\Users\Administrator\fiebaz> ls

    Directory: C:\Users\Administrator\fiebaz

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----         4/9/2019   2:51 PM                hello
d-----         4/9/2019   2:51 PM                tests
-a----       10/26/1985   8:15 AM            230 app.py
-a----       10/26/1985   8:15 AM             32 cdk.json
-a----       10/26/1985   8:15 AM           1375 README.md
-a----       10/26/1985   8:15 AM             12 requirements.txt
-a----       10/26/1985   8:15 AM           1120 setup.py


PS C:\Users\Administrator\fiebaz> python -m venv .env
PS C:\Users\Administrator\fiebaz> .\.env\Scripts\activate
(.env) PS C:\Users\Administrator\fiebaz> pip install -r .\requirements.txt
(.env) PS C:\Users\Administrator\fiebaz> python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import app
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Administrator\fiebaz\app.py", line 3, in <module>
    from aws_cdk import cdk
  File "C:\Users\Administrator\fiebaz\.env\lib\site-packages\aws_cdk\cdk\__init__.py", line 12, in <module>
    import aws_cdk.cx_api
  File "C:\Users\Administrator\fiebaz\.env\lib\site-packages\aws_cdk\cx_api\__init__.py", line 11, in <module>
    __jsii_assembly__ = jsii.JSIIAssembly.load("@aws-cdk/cx-api", "0.28.0", __name__, "[email protected]")
  File "C:\Users\Administrator\fiebaz\.env\lib\site-packages\jsii\_runtime.py", line 40, in load
    _kernel.load(assembly.name, assembly.version, os.fspath(assembly_path))
  File "C:\Users\Administrator\fiebaz\.env\lib\site-packages\jsii\_kernel\__init__.py", line 184, in load
    self.provider.load(LoadRequest(name=name, version=version, tarball=tarball))
  File "C:\Users\Administrator\fiebaz\.env\lib\site-packages\jsii\_kernel\providers\process.py", line 326, in load
    return self._process.send(request, LoadResponse)
  File "C:\Users\Administrator\fiebaz\.env\lib\site-packages\jsii\_utils.py", line 24, in wrapped
    stored.append(fgetter(self))
  File "C:\Users\Administrator\fiebaz\.env\lib\site-packages\jsii\_kernel\providers\process.py", line 321, in _process
    process.start()
  File "C:\Users\Administrator\fiebaz\.env\lib\site-packages\jsii\_kernel\providers\process.py", line 269, in start
    env=environ,
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 1119, in _execute_child
    args = list2cmdline(args)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 530, in list2cmdline
    needquote = (" " in arg) or ("\t" in arg) or not arg
TypeError: argument of type 'WindowsPath' is not iterable

This definitely looks like a bug in the Python JSII kernel when it tries to start the JSII process using the subprocess module. That's nothing like the problems you listed above. It might be useful to try to step through the exact same steps above on your machine to see if you end up in the same place or not. In any case, I'll fix the bug shown above.

[rix0rrr]

Can we (try to) predict whether the correct invocation is python or python3 and act accordingly?

I'd rather fix that as well since even though it's a minor thing, it is bound to confuse and trip up people.

[rix0rrr]

Maybe use this ?

[garnaat]

I agree that fixing the python/python3 problem is also important. I'm not trying to ignore that. I just want to identify and fix any underlying issues in the Python kernel first because until that's fixed nothing will work on Windows.

The py launcher is useful on a Windows machine to find the correct executable but is still Windows-specific and if the user has only Python2 installed the py launcher will just return that. I'm not sure how that helps.

We could try python3 and then, if it fails, try python but we wouldn't know if python really was Python3 or Python2. I guess we could actually run python --version and figure it out.

I kind of feel like the Windows installer for Python is broken if it doesn't install a python3 but even if that's true it doesn't help us.

I don't see a way around doing a bunch of platform-specific stuff in the cdk init code and possibly in the template generation code.

[Doug-AWS]

@garnaat I got the same error you got following your instructions.

[rix0rrr]

From the docs looks like py -3 ought to do it.

And yes, we'd have to switch on the platform. But looks like the python distro from python.org doesn't install python3.exe?

[garnaat]

The above commit fixes the problem I have shown above (the same one @Doug-AWS was able to reproduce). So, progress.

It turns out the shebang #/usr/bin/env python3 works just fine on Windows as-is, even if Python3 is installed as python. The magic of the launcher, I guess.

So, the remaining problem is the explicit calls to python3 in shelled out commands. The cdk init script obviously does this so we have to find a way to figure out the right python command to run rather than hardcoding it. Also, the cdk.json file is hard-coding app: python3 app.py and that needs to change. If you just change this manually to refer to python rather than python3 then things like cdk ls will start working.

In addition, the Python packager in JSII does this all over the place and those explicit calls will fail on Windows and so those, too, need to be platform-aware or packaging will not work on Windows.

I think I will fix the cdk init case here and create another issue for JSII.

[Doug-AWS]

I'm still getting the type error in 0.29.0:

TypeError: argument of type 'WindowsPath' is not iterable

[Doug-AWS]

I missed a step on Windows: .env\Scripts\activate.bat

[AshishGoelTR]

The above commit fixes the problem I have shown above (the same one @Doug-AWS was able to reproduce). So, progress.

It turns out the shebang #/usr/bin/env python3 works just fine on Windows as-is, even if Python3 is installed as python. The magic of the launcher, I guess.

So, the remaining problem is the explicit calls to python3 in shelled out commands. The cdk init script obviously does this so we have to find a way to figure out the right python command to run rather than hardcoding it. Also, the cdk.json file is hard-coding app: python3 app.py and that needs to change. If you just change this manually to refer to python rather than python3 then things like cdk ls will start working.

In addition, the Python packager in JSII does this all over the place and those explicit calls will fail on Windows and so those, too, need to be platform-aware or packaging will not work on Windows.

I think I will fix the cdk init case here and create another issue for JSII.

@garnaat were you able to fix the JSII issue? I am running into the same error. is there a fix out yet?