Can the GUI be updated to use the latest esptool?

[eshattow]

From RTK Firmware Uploader v1.5.2 when used with RTK_Surveyor_Firmware_v3_10.bin:

Detecting flash size


______________________________________________________________________
ESP32 Flash Size Detection

Command:	['--chip', 'esp32', '--port', '/dev/ttyUSB0', '--baud', '460800', '--before', 'default_reset', '--after', 'no_reset', 'flash_id']

esptool.py v3.3
Serial port /dev/ttyUSB0
Connecting......................................

Flash detection complete. Uploading firmware...
Flash size not detected! Defaulting to 16MB

Uploading firmware

______________________________________________________________________
ESP32 Firmware Upload

Command:	['--chip', 'esp32', '--port', '/dev/ttyUSB0', '--baud', '460800', '--before', 'default_reset', '--after', 'no_reset', 'write_flash', '-z', '--flash_mode', 'dio', '--flash_freq', '80m', '--flash_size', 'detect', '0x1000', '/home/es/software/pubscm/SparkFun_RTK_Firmware_Uploader.git/RTK_Firmware_Uploader/resource/RTK_Surveyor.ino.bootloader.bin', '0x8000', '/home/es/software/pubscm/SparkFun_RTK_Firmware_Uploader.git/RTK_Firmware_Uploader/resource/RTK_Surveyor_Partitions_16MB.bin', '0xe000', '/home/es/software/pubscm/SparkFun_RTK_Firmware_Uploader.git/RTK_Firmware_Uploader/resource/boot_app0.bin', '0x10000', '/home/es/software/pubscm/SparkFun_RTK_Firmware_Binaries.git/RTK_Surveyor_Firmware_v3_10.bin']

esptool.py v3.3
Serial port /dev/ttyUSB0
Connecting......................................

Firmware upload complete. Resetting ESP32...
Resetting ESP32

______________________________________________________________________
ESP32 Reset

Command:	['--chip', 'esp32', '--port', '/dev/ttyUSB0', '--before', 'default_reset', 'run']

esptool.py v3.3
Serial port /dev/ttyUSB0
Connecting......................................

Reset complete...

I've tried all three listed baud rates (115200, 460800, 921600) but the outcome is the same. See sparkfun/SparkFun_RTK_Firmware#361 and sparkfun/SparkFun_RTK_Firmware#266 it is the same hardware. Is there no way to do this with the GUI?

Additionally I tried:

SparkFun_RTK_Firmware_Uploader.git/RTK_Firmware_Uploader$ sudo python3 esptool.py --chip esp32 --port /dev/ttyUSB0 --baud 921600 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size detect 0x1000 resource/RTK_Surveyor.ino.bootloader.bin 0x8000 resource/RTK_Surveyor_Partitions_4MB.bin 0xe000 resource/boot_app0.bin 0x10000 ../../SparkFun_RTK_Firmware_Binaries.git/RTK_Surveyor_Firmware_v3_10.bin 
esptool.py v3.3
Serial port /dev/ttyUSB0
Connecting......................................

A fatal error occurred: Failed to connect to ESP32: Wrong boot mode detected (0x13)! The chip needs to be in download mode.
For troubleshooting steps visit: https://docs.espressif.com/projects/esptool/en/latest/troubleshooting.html

[eshattow]

Attempted to follow more closely the instructions of https://github.com/sparkfun/SparkFun_RTK_Firmware/blob/main/docs/firmware_update.md with some success but the GUI does not.

Cloning https://github.com/espressif/esptool.git and running esptool from that with chip_id I get:

esptool.git$ python3 esptool.py flash_id
esptool.py v4.7-dev
Found 1 serial ports
Serial port /dev/ttyUSB0
Connecting............
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting......
Detecting chip type... ESP32
Chip is ESP32-D0WDQ6 (revision v1.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: ac:67:b2:1b:0d:18
Uploading stub...
Running stub...
Stub running...
Manufacturer: 20
Device: 4016
Detected flash size: 4MB
Hard resetting via RTS pin...

Where it is written in firmware_update.md:

esptool.exe --chip esp32 --port COM6 --baud 921600 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size detect 0x1000 ./bin/RTK_Surveyor.ino.bootloader.bin 0x8000 ./bin/RTK_Surveyor_Partitions_4MB.bin 0xe000 ./bin/boot_app0.bin 0x10000 ./RTK_Surveyor_Firmware_vxx.bin

I have transliterated that to instead run the following:

esptool.git$ python3 esptool.py --chip esp32 --port /dev/ttyUSB0 --baud 921600 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size detect 0x1000 ../SparkFun_RTK_Firmware_Uploader.git/RTK_Firmware_Uploader/resource/RTK_Surveyor.ino.bootloader.bin 0x8000 ../SparkFun_RTK_Firmware_Uploader.git/RTK_Firmware_Uploader/resource/RTK_Surveyor_Partitions_4MB.bin 0xe000 ../SparkFun_RTK_Firmware_Uploader.git/RTK_Firmware_Uploader/resource/boot_app0.bin 0x10000 ../SparkFun_RTK_Firmware_Binaries.git/RTK_Surveyor_Firmware_v3_10.bin 
esptool.py v4.7-dev
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP32-D0WDQ6 (revision v1.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: ac:67:b2:1b:0d:18
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Flash will be erased from 0x00001000 to 0x00005fff...
Flash will be erased from 0x00008000 to 0x00008fff...
Flash will be erased from 0x0000e000 to 0x0000ffff...
Flash will be erased from 0x00010000 to 0x0023afff...
Compressed 17984 bytes to 12318...
Wrote 17984 bytes (12318 compressed) at 0x00001000 in 0.4 seconds (effective 368.9 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 119...
Wrote 3072 bytes (119 compressed) at 0x00008000 in 0.0 seconds (effective 562.4 kbit/s)...
Hash of data verified.
Compressed 8192 bytes to 47...
Wrote 8192 bytes (47 compressed) at 0x0000e000 in 0.1 seconds (effective 728.3 kbit/s)...
Hash of data verified.
Compressed 2269328 bytes to 1493800...
Wrote 2269328 bytes (1493800 compressed) at 0x00010000 in 24.4 seconds (effective 744.1 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

confirmed via serial output:

Batt (96%): Voltage: 4.19V Charging: 7.70%/hr Green

SparkFun RTK Surveyor v3.10

...and so that seems to be functional. Does esptool for the GUI need to be updated? Aside I had to pull in apt install python3-intelhex on this Debian host to run the esptool.py from esptool git head.

[PaulZC]

Hi @eshattow ,

Apologies for the inconvenience and your long history of wrestling with RTK updates on Linux. Thank you for the detective work. If esptool v4.7 fixes this issue then we will absolutely update the GUI to match. I'm looking at it now.

Best wishes,
Paul

[PaulZC]

Hi @eshattow ,

I'm struggling with this...

I've created a new v1.6.0 branch. In there, I've deleted the old copy of esptool.py and instead have the build workflow install the latest release of esptool (4.6.2) manually.

I'm on Windows. If I run RTK_Firmware_Upload.py locally, having first pip install esptool, everything works:

But if I run the executable created by the workflow, it fails silently at "Uploading stub...":

Interestingly, if I leave the executable open, and re-run the uploader locally, it then fails at the same place. But starts working as soon as I close the executable.

Something is being loaded differently between the two environments. I suspect (py)serial, but I don't yet know how to prove that or correct it.

Thanks for the pointer about intelhex. The build workflow seemed to be missing it - even though esptool lists it as a dependency. I changed the workflows to install it manually, but that didn't fix this issue.

I don't think this is related to this issue, but it could be.

I'll have to give up on this for now and will come back to it on Monday.

More later,
Paul

[PaulZC]

I now suspect this is caused by argparse but I am struggling to prove it. I think the no-stub arg is not being defined (set to false). Very strange...

[PaulZC]

Ah. Cancel that. I now suspect it is because it can't find stub_flasher_32.json. I bet that isn't being added to the pyinstaller installer...?

[PaulZC]

The same issue is described here: espressif/esptool#208 (comment)

OK. I'm giving up on this. Sorry! Generating a self-consistent GUI executable seems very difficult with esptool ~4.6.2.

You can download the GUI source here: https://github.com/sparkfun/SparkFun_RTK_Firmware_Uploader/archive/refs/heads/v1.6.0.zip
Install esptool manually using pip install esptool
Run the GUI manually with python RTK_Firmware_Upload.py
It will import esptool, find the json files and all will be well...

[eshattow]

Note: esptool 4.7.0 released upstream a couple weeks ago.

[PaulZC]

Thanks for the heads-up @eshattow

Humm... I wonder if there is a sneaky way to add the stub_flasher json files to the pyinstaller installer?
Reopening this...

[PaulZC]

Hi @eshattow ,

I've made some progress with this: I have it working nicely on Windows; I can't get it to work on macOS; and I don't have a Linux machine to test it on.

There were two 'tricks' to get it running on Windows:

• Add the esptool/targets/stub_flasher json files to the pyinstaller package so the .pyc can find them
• Modify the RTK_Surveyor.ino.bootloader.bin and set the Extended File Header Hash Appended byte to zero
  • RTK_Surveyor.ino.bootloader.bin has the File Header flash mode, size and frequency set to 0x22f (DIO, 4MB, 80MHz). That's fine for the original 4MB RTK Surveyor, but not OK for the later 16MB products. The size nibble needs to be changed to 4 (0x24f).
  • It also has the Hash Appended byte set to 1
  • The older version of esptool (~3.3) ignored the Hash Appended byte and changed the size and frequency as needed
  • The latest version of esptool (4.7) checks the Hash Appended byte and will only modify the size and frequency if it is zero
  • The best solution would be to produce 4MB and 16MB versions of the bootloader with correct hash's
  • The work-around it to manually set that byte to zero using a hex editor

This works on Windows, but fails on macOS. It appears that the esptool pyc still can't find the stub_flasher json files.

Best wishes,
Paul

[eshattow]

RTK Surveyor 4MB is attached and powered on.

$ ./RTKUploader.linux

Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway.
/usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so: undefined symbol: g_task_set_static_name
Failed to load module: /usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so

** (RTKUploader.linux:83983): WARNING **: 11:49:54.281: atk-bridge: get_device_events_reply: unknown signature

Choose Firmware File Browse to /tmp/SparkFun_RTK_Firmware_Binaries.git/RTK_Surveyor_Firmware_v4_0.bin

/usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so: undefined symbol: g_task_set_static_name
Failed to load module: /usr/lib/x86_64-linux-gnu/gio/modules/libgioremote-volume-monitor.so
/usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so: undefined symbol: g_task_set_static_name
Failed to load module: /usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so
/usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so: undefined symbol: g_task_set_static_name
Failed to load module: /usr/lib/x86_64-linux-gnu/gio/modules/libgioremote-volume-monitor.so

(RTKUploader.linux:83983): Gtk-WARNING **: 11:50:27.478: /usr/lib/x86_64-linux-gnu/gtk-3.0/3.0.0/immodules/im-wayland.so: undefined symbol: g_string_free_and_steal

(RTKUploader.linux:83983): Gtk-WARNING **: 11:50:27.478: Loading IM context type 'wayland' failed

(RTKUploader.linux:83983): Gtk-WARNING **: 11:50:27.488: /usr/lib/x86_64-linux-gnu/gtk-3.0/3.0.0/immodules/im-wayland.so: undefined symbol: g_string_free_and_steal

(RTKUploader.linux:83983): Gtk-WARNING **: 11:50:27.488: Loading IM context type 'wayland' failed

(RTKUploader.linux:83983): Gtk-WARNING **: 11:50:27.489: /usr/lib/x86_64-linux-gnu/gtk-3.0/3.0.0/immodules/im-wayland.so: undefined symbol: g_string_free_and_steal

(RTKUploader.linux:83983): Gtk-WARNING **: 11:50:27.489: Loading IM context type 'wayland' failed

(RTKUploader.linux:83983): Gtk-WARNING **: 11:50:27.490: /usr/lib/x86_64-linux-gnu/gtk-3.0/3.0.0/immodules/im-wayland.so: undefined symbol: g_string_free_and_steal

(RTKUploader.linux:83983): Gtk-WARNING **: 11:50:27.490: Loading IM context type 'wayland' failed

(RTKUploader.linux:83983): Gtk-WARNING **: 11:50:40.968: /usr/lib/x86_64-linux-gnu/gtk-3.0/3.0.0/immodules/im-wayland.so: undefined symbol: g_string_free_and_steal

(RTKUploader.linux:83983): Gtk-WARNING **: 11:50:40.968: Loading IM context type 'wayland' failed

Upload Firmware

Detecting flash size


______________________________________________________________________
ESP32 Flash Size Detection

Command:	['--chip', 'esp32', '--port', '/dev/ttyUSB0', '--baud', '921600', '--before', 'default_reset', '--after', 'no_reset', 'flash_id']

esptool.py v4.7.0
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP32-D0WDQ6 (revision v1.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: ac:67:b2:1b:0d:18
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Manufacturer: 20
Device: 4016
Detected flash size: 4MB
Staying in bootloader.

Flash detection complete. Uploading firmware...
Flash size is 4MB

Uploading firmware

______________________________________________________________________
ESP32 Firmware Upload

Command:	['--chip', 'esp32', '--port', '/dev/ttyUSB0', '--baud', '921600', '--before', 'default_reset', '--after', 'no_reset', 'write_flash', '-z', '--flash_mode', 'dio', '--flash_freq', '80m', '--flash_size', 'detect', '0x1000', '/tmp/_MEIQIvtiB/resource/RTK_Surveyor.ino.bootloader.bin', '0x8000', '/tmp/_MEIQIvtiB/resource/RTK_Surveyor_Partitions_4MB.bin', '0xe000', '/tmp/_MEIQIvtiB/resource/boot_app0.bin', '0x10000', '/tmp/SparkFun_RTK_Firmware_Binaries.git/RTK_Surveyor_Firmware_v4_0.bin']

esptool.py v4.7.0
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP32-D0WDQ6 (revision v1.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: ac:67:b2:1b:0d:18
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Flash will be erased from 0x00001000 to 0x00005fff...
Flash will be erased from 0x00008000 to 0x00008fff...
Flash will be erased from 0x0000e000 to 0x0000ffff...
Flash will be erased from 0x00010000 to 0x0023bfff...
Compressed 17984 bytes to 12317...
Writing at 0x00001000... (100 %)
Wrote 17984 bytes (12317 compressed) at 0x00001000 in 0.4 seconds (effective 366.6 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 119...
Writing at 0x00008000... (100 %)
Wrote 3072 bytes (119 compressed) at 0x00008000 in 0.0 seconds (effective 541.5 kbit/s)...
Hash of data verified.
Compressed 8192 bytes to 47...
Writing at 0x0000e000... (100 %)
Wrote 8192 bytes (47 compressed) at 0x0000e000 in 0.1 seconds (effective 725.7 kbit/s)...
Hash of data verified.
Compressed 2275472 bytes to 1497673...
Writing at 0x00010000... (1 %)
Writing at 0x0001ba5b... (2 %)
Writing at 0x00021a89... (3 %)
Writing at 0x00025aef... (4 %)
Writing at 0x00029aea... (5 %)
Writing at 0x0002dae5... (6 %)
Writing at 0x00031b28... (7 %)
Writing at 0x00035bc6... (8 %)
Writing at 0x00039bb2... (9 %)
Writing at 0x0003dbad... (10 %)
Writing at 0x00041df6... (11 %)
Writing at 0x00048fe1... (13 %)
Writing at 0x0005668f... (14 %)
Writing at 0x0006058b... (15 %)
Writing at 0x00069d84... (16 %)
Writing at 0x00072f01... (17 %)
Writing at 0x000870e9... (18 %)
Writing at 0x00093f66... (19 %)
Writing at 0x00099ea6... (20 %)
Writing at 0x0009f659... (21 %)
Writing at 0x000a527b... (22 %)
Writing at 0x000aad8d... (23 %)
Writing at 0x000b073f... (25 %)
Writing at 0x000b5e1f... (26 %)
Writing at 0x000bbcc6... (27 %)
Writing at 0x000c1664... (28 %)
Writing at 0x000c7916... (29 %)
Writing at 0x000cd233... (30 %)
Writing at 0x000d2f74... (31 %)
Writing at 0x000d87e7... (32 %)
Writing at 0x000de76f... (33 %)
Writing at 0x000e4a93... (34 %)
Writing at 0x000ea693... (35 %)
Writing at 0x000f044f... (36 %)
Writing at 0x000f62de... (38 %)
Writing at 0x000fb690... (39 %)
Writing at 0x001007f3... (40 %)
Writing at 0x0010629a... (41 %)
Writing at 0x0010bf4d... (42 %)
Writing at 0x00111464... (43 %)
Writing at 0x00116b08... (44 %)
Writing at 0x0011bef8... (45 %)
Writing at 0x0012115c... (46 %)
Writing at 0x0012651e... (47 %)
Writing at 0x0012c57b... (48 %)
Writing at 0x00131fbf... (50 %)
Writing at 0x0013762a... (51 %)
Writing at 0x0013ca11... (52 %)
Writing at 0x00141e1a... (53 %)
Writing at 0x001481f1... (54 %)
Writing at 0x0014da61... (55 %)
Writing at 0x001531b3... (56 %)
Writing at 0x00158eb6... (57 %)
Writing at 0x0015e795... (58 %)
Writing at 0x0016411d... (59 %)
Writing at 0x00169974... (60 %)
Writing at 0x0016f0ee... (61 %)
Writing at 0x00174bb2... (63 %)
Writing at 0x0017a4cd... (64 %)
Writing at 0x0017fb5c... (65 %)
Writing at 0x00185555... (66 %)
Writing at 0x0018a7be... (67 %)
Writing at 0x0018f909... (68 %)
Writing at 0x00195176... (69 %)
Writing at 0x0019a313... (70 %)
Writing at 0x001a0107... (71 %)
Writing at 0x001a6586... (72 %)
Writing at 0x001ac35a... (73 %)
Writing at 0x001b14c1... (75 %)
Writing at 0x001b6c71... (76 %)
Writing at 0x001bcef6... (77 %)
Writing at 0x001c2381... (78 %)
Writing at 0x001c7951... (79 %)
Writing at 0x001cd0ef... (80 %)
Writing at 0x001d23ad... (81 %)
Writing at 0x001d7777... (82 %)
Writing at 0x001dcb6c... (83 %)
Writing at 0x001e22a0... (84 %)
Writing at 0x001e7a57... (85 %)
Writing at 0x001ed833... (86 %)
Writing at 0x001f3253... (88 %)
Writing at 0x001f8a9c... (89 %)
Writing at 0x00201736... (90 %)
Writing at 0x00209f23... (91 %)
Writing at 0x00210c74... (92 %)
Writing at 0x00215fad... (93 %)
Writing at 0x0021da3d... (94 %)
Writing at 0x00223140... (95 %)
Writing at 0x002286ff... (96 %)
Writing at 0x0022e556... (97 %)
Writing at 0x002336f8... (98 %)
Writing at 0x0023928b... (100 %)
Wrote 2275472 bytes (1497673 compressed) at 0x00010000 in 24.4 seconds (effective 745.1 kbit/s)...
Hash of data verified.

Leaving...
Staying in bootloader.

Firmware upload complete. Resetting ESP32...
Resetting ESP32

______________________________________________________________________
ESP32 Reset

Command:	['--chip', 'esp32', '--port', '/dev/ttyUSB0', '--before', 'default_reset', 'run']

esptool.py v4.7.0
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP32-D0WDQ6 (revision v1.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: ac:67:b2:1b:0d:18
Uploading stub...
Running stub...
Stub running...
Hard resetting via RTS pin...

Reset complete...