RP2040: Windows build (was QEMU emulator)

[wilberforce]

I looked at ordering... here in NZ it looks like I won't be able to get until end of March/April.

Can it be run in QEMU or an emulator?

I guess the next steps for the port will be IO type things - so real hardware would be needed.

[phoddie]

That's a really long wait.

Can it be run in QEMU or an emulator?

Good question. I haven't seen anything about that.

If I recall well, you usually work on Windows. There's no Windows build in the Moddable SDK for Pico at the moment. That could certainly be done without the hardware. Someone else could help test the output.

[wilberforce]

This looks like a useful start

https://github.com/ndabas/pico-setup-windows

[wilberforce]

@andycarle
The github link above is a great example of how to wrap up the command line tools - you could make an installer for windows esp32 and have an installer exe - very slick

[wilberforce]

This is where I'm at:

update moddable

git fetch
git pull upstream pico

rebuild...

cd %MODDABLE%\build\makefiles\win
build clean
build

download Pico framework installer:

https://github.com/ndabas/pico-setup-windows
install to %USERPROFILE%\Pico

This will download all the tools locally and build the examples.

Attempt to build moddable helloworld:

cd %USERPROFILE%\Pico
open Developer Command Prompt for Pico

cd %USERPROFILE%\Projects\moddable\examples\helloworld

mcconfig -d -m -p pico

### Error: unknown platform: MAKE_FRAGMENT not found!

Not sure what the cause of this is at this stage.

C:\Users\rhys\Projects\moddable\examples\helloworld>dir ..\..\build\devices\pico

 Directory of C:\Users\rhys\Projects\moddable\build\devices\pico

21/02/2021  02:58 PM    <DIR>          .
21/02/2021  02:58 PM    <DIR>          ..
21/02/2021  02:58 PM    <DIR>          base
21/02/2021  02:58 PM    <DIR>          config
21/02/2021  02:58 PM               388 manifest.json

So the pico is in the devices folder!

[andycarle]

Nice that you got everything installed! But, yes, you've reached what is likely to be the most challenging step of the Windows bring-up.

Each of the Moddable SDK target platforms has two Makefile fragments associated with it. A make.<target>.mk for macOS and Linux and a nmake.<target>.mk for Windows. For instance, there are make.esp32.mk and nmake.esp32.mk.

You will see that in the pico branch there is currently a make.pico.mk, but no nmake.pico.mk to go with it.

About half of the process of creating the nmake version is straightforward. Examples: Sources, include paths, etc. usually just need to have \ swapped in for /.... Other paths need to be adjusted... Linux-style commands need to be replaced with their Windows equivalents (e.g. cp -> copy) with fixed arguments/flags... etc.

The challenging half comes from the fact that the make.<target>.mk fragments use GNU make extensions that are not available to NMAKE. Some of those are simple (e.g. GNU make has ?= for "if undefined, equals" while that needs to be spelled out with a full conditional in the NMAKE version). Some of them are more challenging (no shell command in NMAKE, no call in NMAKE, no foreach in NMAKE, etc.) I often get around these challenges by writing supplemental batch files or (in the case of the ESP32 build), some tiny Python scripts (which I consider fair game because the ESP-IDF build requires Python anyways). But it is definitely a process to work through it all.

If no one else does so, I will take the first pass on putting nmake.pico.mk together early this coming week. But I would certainly welcome help!

[wilberforce]

@andycarle

Nice that you got everything installed!
https://github.com/ndabas/pico-setup-windows

This made it so easy - I just installed with all the options.

For instance, there are make.esp32.mk and nmake.esp32.mk.
thanks - It's pointed me in the right direction...

I have made some progress... I have got to the point where nmake is running.... although there are a lot of:
# To Do where I don't know the nmake equivalent.

nmake.pico.mk.zip

It's currently barfing with:

C:\Users\rhys\Projects\moddable\examples\helloworld>mcconfig -d -m -p pico
NMAKE : fatal error U1073: don't know how to make 'C:\Users\rhys\Projects\moddable\build\tmp\pico\debug\helloworld\xsmain.c.o'
Stop.

There seems to be rule... however it might be a nmake vs make thing!

[wilberforce]

@andycarle
I've expanded this macro.... still not finding xsmain and other tools: ( around line 619)

#C_INCLUDES = $(foreach dir,$(INC_DIRS) $(SDK_GLUE_DIRS) $(XS_DIRS) $(LIB_DIR) $(TMP_DIR),-I$(call qs,$(dir)))  $(C_INCLUDES)
C_INCLUDES = $(C_INCLUDES) $(DIRECTORIES) -I$(INC_DIRS)  -I$(SDK_GLUE_DIRS) -I$(XS_DIRS) -I$(LIB_DIR) -I$(TMP_DIR)

I see the nmake.esp32.mk has these kind of rules:

$(PROJ_DIR)\main\main.c: $(PROJ_DIR_TEMPLATE)\main\main.c
copy $? $@

I guess the SDK_GLUE_DIRS files need to be copied to the dest build folder.... ?

[andycarle]

Headed in the right direction!

With GNU Make, ${TMP_DIR)/xsmain.c.o is being built by this rule, I believe:

$(TMP_DIR)\%.c.o: %.c
	@echo "# application: " $(<F)
	$(CC) $(C_FLAGS) $(C_INCLUDES) $(C_DEFINES) $< -o $@

That is GNU Make extension syntax that will automatically match requirements to targets with wildcards, such that any requirement of the form ${TMP_DIR}\*.c.o will match to that target/rule and it will find the corresponding source file. NMAKE does not exactly support wildcards in targets like this, unfortunately.

One fix is to just have a separate rule for each file. But, NMAKE does support a per-directory wildcard syntax. For example, from nmake.esp32.mk:

{$(XS_DIR)\platforms\esp\}.c{$(LIB_DIR)\}.o:
	@echo # cc $(@F) (strings in flash)
	$(CC) $(C_DEFINES) $(C_INCLUDES) $(C_FLAGS) $< -o $@

Something similar could be used to build the object files from the various SDK_GLUE_DIRS directories.

[wilberforce]

more progress.... one step forward, two steps back....

nmake.pico.mk.zip

Currently the output is:

....co\debug\helloworld\xsmain.c -o C:\Users\rhys\Projects\moddable\build\tmp\pico\debug\helloworld\xsmain.c.o
In file included from C:\Users\rhys\Projects\moddable\xs\platforms\pico/xsHost.h:44,
                 from C:\Users\rhys\Projects\moddable\xs\platforms\pico/xsPlatform.h:93,
                 from C:\Users\rhys\Projects\moddable\build\tmp\pico\debug\helloworld\xsmain.h:24,
                 from C:\Users\rhys\Projects\moddable\build\tmp\pico\debug\helloworld\xsmain.c:22:
C:\Users\rhys\Pico\pico-sdk\src\common\pico_stdlib\include/pico/stdlib.h:10:10: fatal error: pico.h: No such file or directory
   10 | #include "pico.h"
      |          ^~~~~~~~

line 637 - I can't seem to include all of those files as the line gets too long to execute.

[phoddie]

@mkellner, you have recent experience with long command lines on Windows. Any advice?

[mkellner]

Hi!
For a different project, I was able to reduce the problem with a couple of changes:

• Move moddable somewhere closer to root. Like MODDABLE=c:\moddable
• Take advantage of the -iwithprefix option to define subsets of paths
  It would look something like:

INC_DIRS=-iprefix $(PICO_SDK_DIR)\src\common \
    -iwithprefix pico_stdlib\include \
    -iwithprefix pico_base\include \
    -iwithprefix pico_time\include \
    -iwithprefix pico_sync\include \
...
    -iprefix $(PICO_SDK_DIR)\src\rp2_common \
    -iwithprefix hardware_gpio\include \
    -iwithprefix hardware_pwm\include \
    -iwithprefix hardware_base\include \

[wilberforce]

@phoddie

I have got further and the compile is starting. Yay!

There is an issue - I think there might be a file missing:

C:\Users\rhys\Projects\moddable\build\tmp\pico\debug\helloworld\debugger.c:26:10: fatal error: tusb.h: No such file or directory
26 | #include "tusb.h"
| ^~~~~~~~

Thanks!

edit looks like it's part of tinyusb.... I'll need to include...

[mkellner]

tusb.h is in the pico-sdk/lib/tinyusb/tusb.h
Is it in your pico-sdk?
If not, did you do the git submodule update --init in the pico-sdk directory?

[wilberforce]

I have that working now.. - thanks

[wilberforce]

Ok... building further now.

It's failing trying to build the pico sdk files...

NMAKE : fatal error U1073: don't know how to make 'C:\Users\rhys\Projects\moddable\build\tmp\pico\debug\lib\gpio.o'

I have copied the files to the tmp folder with:

FOR /R $(PICO_SDK_DIR)\src %G IN (*.c) DO copy %G $(LIB_DIR)

nmake.pico.mk.zip

The zip contains the nmake file and the output.

I have managed to the first in the sdk with an implicit rule:

$(LIB_DIR)\stdlib.o: $(LIB_DIR)\stdlib.c
	$(CC) $(C_DEFINES) $(C_INCLUDES) $(C_FLAGS) $? -o $@

This rule should match - but doesn't

# TO NOT WORKING..
{$(LIB_DIR)\}.c{$(LIB_DIR)\}.o:
	$(CC) $(C_DEFINES) $(C_INCLUDES) $(C_FLAGS) $< -o $@

I had to change the .c.o to .o as the rule would not allow .c.o - the esp32 nmake seems to follow this pattern too..

[mkellner]

In the nmake.esp32.mk, I see:

{$(XS_DIR)\sources\}.c{$(LIB_DIR)\}.o:
    @echo # cc $(@F) (strings in flash)
    $(CC) $(C_DEFINES) $(C_INCLUDES) $(C_FLAGS) $< -o $@

Where the source-file path/wildcard is quite different from the destination path.

The unworking one above has the source and destination directory the same. Is that intentional?
{$(LIB_DIR)\}.c{$(LIB_DIR)\}.o:

The Moddable SDK defines and uses TMP_DIR, LIB_DIR and BIN_DIR in the build. Have you re-defined LIB_DIR?

[mkellner]

I do see that there is a buildinfo.h file that is created into $(LIB_DIR)\buildinfo.c, so {$(LIB_DIR)\}.c{$(LIB_DIR)\}.o: may be fine - but there is already a build target for $(LIB_DIR)\buildinfo.c.o.

[phoddie]

Check out this forum thread. It does look like many people are wrestling with Pico builds on Windows. I didn't realize how strongly the Pico is a Linux-centric device for development.

[wilberforce]

more progress - 57.o files are building now...

nmake.pico.mk.zip

current failure point:

modTime.c
C:\Users\rhys\Pico\pico-sdk\src\common\pico_stdlib\include\pico/stdlib.h(10): fatal error C1083: Cannot open include file: 'pico.h': No such file or directory

I have ended up hardcoding each SDK source .c as there are in different folders and could not the the { } substitute to work.

[phoddie]

Strange that pico.h is choosing to be difficult. I would expect to have already been included by some of the 57 (!) files that do build. After all pico.h has this comment:

This header is intended to be included by all source code

[wilberforce]

@andycarle @phoddie

Talking a bit of a fresh look at this - since nmake is a a bit of a maze - is there any reason why we can't using gmake and the normal make files to build a project like this?

[phoddie]

Does the default Pico build on Windows use cmake?

[wilberforce]

The pico-sdk itself uses cmake. The mcconfig for windows for the pcio target is set up to use nmake

[phoddie]

Interesting.

[andycarle]

It's a fair question, and one that I return to pretty frequently myself when mired in NMAKE arcana.

It has been a while (like... this was at the previous company) since I've tried doing builds using Make for Windows. It didn't go well at the time, but it's worth another look. Testing it will require some tweaks to the mcconfig sources, but I think I can try that this week. (This also is generally not the right approach for building the native bits, like tools and simulators. But it could work for the cross-compilation targets.)

If I'm remembering correctly, Make for Windows works via MinGW. That's not inherently bad, but it does add some complications around paths and it can introduce some slowness. There's a reason I chased it out of the ESP32 build. :)

As to CMake, I like it, personally. And as you know, it is used extensively by idf.py in the ESP32 builds on all platforms, so our developers often already have it installed. But it would be a substantial project (both in implementation and maintenance) to change mcconfig and friends to generate CMake fragments instead of Make fragments. (If I'm remembering right,, we've been down that road once before, also at the previous company...) I don't see that happening imminently.

But there might be some middle ground to be explored here, where CMake is used to generate the host platform-specific Makefile fragment before mcconfig incorporates the fragment into the project Makefile. Probably also worth looking at.

[wilberforce]

@andycarle I'm not suggesting changing the win tools from nmake. That all works as is. I think for each target is the place where simplification can occur. Both pico and esp-idf using cmake under the hood, and the gnu tools set - which also includes the gnu make. If the platform tool set build where ported to use cmake, then gnu make could be the makefile target for each. This would simplify the toolchain for the targets, and avoid complexities like being MinGW etc.