[Bug Report]: (Mini|Ana)Conda virtual python environments almost work (VSC-516)

[brainstorm]

Pre Bug Report Checklist
Before reporting any bug please make sure of these points.

• Make sure you have searched for existing bugs and features request before you post an issue.
• This is a bug report for the ESP-IDF Visual Studio Code extension and not an ESP-IDF bug report.
• I've read the docs and found no information that could have helped solving the issue.

Describe the bug

Since I use Anaconda venvs for my regular work instead of python3 venvs, I'm prefixing the tasks with conda activate espressif (a self-contained venv for this extension), like this:

        {
            "label": "Flash - Flash the device",
            "type": "shell",
            "command": "conda activate espressif && ${config:idf.pythonBinPath} ${config:idf.espIdfPath}/tools/idf.py -p ${config:idf.port} -b ${config:idf.baudRate} flash",
            "windows": {
                "command": "${config:idf.pythonBinPathWin} ${config:idf.espIdfPathWin}\\tools\\idf.py flash -p ${config:idf.portWin} -b ${config:idf.baudRate}"

Compiling and flashing works fine, but unfortunately, monitor does not follow the same logic since it fails miserably on this setup:

export IDF_PATH=/Users/romanvg/dev/esp-idf
python /Users/romanvg/dev/esp-idf/tools/idf.py -p /dev/cu.SLAB_USBtoUART monitor

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
(base) brainstorm:twai_network_slave romanvg$ export IDF_PATH=/Users/romanvg/dev/esp-idf
(base) brainstorm:twai_network_slave romanvg$ python /Users/romanvg/dev/esp-idf/tools/idf.py -p /dev/cu.SLAB_USBtoUART monitor
Traceback (most recent call last):
  File "/Users/romanvg/dev/esp-idf/tools/idf.py", line 46, in <module>
    from idf_py_actions.tools import (executable_exists, idf_version, merge_action_lists, realpath)  # noqa: E402
  File "/Users/romanvg/dev/esp-idf/tools/idf_py_actions/tools.py", line 1, in <module>
    import click
ImportError: No module named click
(base) brainstorm:twai_network_slave romanvg$ 

The click dependency is installed fine under my espressif conda venv:

$ pip freeze | grep click
click==7.1.2

Your direct competitor (partner?), PlatformIO, fixed Conda environments a while ago, see also: platformio/platformio-atom-ide#218 (comment)

To Reproduce

1. Install Miniconda.
2. Setup this extension in VSCode.
3. Run monitor buttton.

Environment (please complete the following information):

• OS Name & Version: MacOS X 10.15.2 (Catalina)
• VSCode Version: Insiders, latest
• ESP-IDF Version: master
• Python Version: 3.9.1

[brainstorm]

Could it be that createMonitor() just injects commands in a new VSCode terminal instead of running a task (like the rest compile/flash/etc tasks do?), see, for OSX:

vscode-esp-idf-extension/src/extension.ts

Line 2373 in 98cc4d2

monitorTerminal.sendText(`${pythonBinPath} ${idfPath} -p ${port} monitor`);

That would explain the garbled $PATH and the conda activate espressif prefix on the task not being respected :/

[brainstorm]

Running it manually for a VSCode terminal with the venv loaded works fine:

(espressif) brainstorm:twai_network_slave romanvg$ python /Users/romanvg/dev/esp-idf/tools/idf.py -p /dev/cu.SLAB_USBtoUART monitor

Setting IDF_PATH environment variable: /Users/romanvg/dev/esp-idf
Executing action: monitor
Running idf_monitor in directory /Users/romanvg/dev/esp32s2_can/twai_network_slave
Executing "/Users/romanvg/.miniconda3/envs/espressif/bin/python /Users/romanvg/dev/esp-idf/tools/idf_monitor.py -p /dev/cu.SLAB_USBtoUART -b 115200 --toolchain-prefix xtensa-esp32-elf- /Users/romanvg/dev/esp32s2_can/twai_network_slave/build/twai_network_slave.elf -m '/Users/romanvg/.miniconda3/envs/espressif/bin/python' '/Users/romanvg/dev/esp-idf/tools/idf.py' '-p' '/dev/cu.SLAB_USBtoUART'"...
--- idf_monitor on /dev/cu.SLAB_USBtoUART 115200 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
device reports readiness to read but returned no data (device disconnected or multiple access on port?)
EWaiting for the device to reconnect
ESP-ROM:esp32s2-rc4-20191025
Build:Oct 25 2019
rst:0x7 (TG0WDT_SYS_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x4000f79a
Invalid chip id. Expected 2 read 0. Bootloader for wrong chip?
ets_main.c 386 
ESP-ROM:esp32s2-rc4-20191025
Build:Oct 25 2019
rst:0x10 (RTCWDT_RTC_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
Invalid chip id. Expected 2 read 0. Bootloader for wrong chip?
ets_main.c 386 
ESP-ROM:esp32s2-rc4-20191025
Build:Oct 25 2019
(...)

Well, I need to change the chip ID to something for the ESP32S2 specifically, but I hope you get the point ;)

EDIT: idf.py set-target esp32s2 did not help but that's off-topic... or perhaps a dedicated VSCode palette command such as ESP-IDF: Select espressif target IC would be great? Will open another issue for this.

[brainstorm]

... and now after the 0.6.0 update all tasks stopped working altogether, without error nor general status outputs only a brief "Building/Flashing" popup shows up and vanishes in 1-2 seconds without followup :/

OTOH all commands work via console (after removing CMakeCache.txt in the case of the build task):

$ ~/dev/esp-idf/tools/idf.py build
Executing action: all (aliases: build)
Running cmake in directory /Users/romanvg/dev/esp32s2_can/twai_network_slave/build
Executing "cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DESP_PLATFORM=1 -DIDF_TARGET=esp32s2 -DCCACHE_ENABLE=0 /Users/romanvg/dev/esp32s2_can/twai_network_slave"...
-- Found Git: /usr/local/bin/git (found version "2.25.1") 
-- The C compiler identification is GNU 8.4.0
-- The CXX compiler identification is GNU 8.4.0
-- The ASM compiler identification is GNU
-- Found assembler: /Users/romanvg/.espressif/tools/xtensa-esp32s2-elf/esp-2020r3-8.4.0/xtensa-esp32s2-elf/bin/xtensa-esp32s2-elf-gcc
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Users/romanvg/.espressif/tools/xtensa-esp32s2-elf/esp-2020r3-8.4.0/xtensa-esp32s2-elf/bin/xtensa-esp32s2-elf-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Users/romanvg/.espressif/tools/xtensa-esp32s2-elf/esp-2020r3-8.4.0/xtensa-esp32s2-elf/bin/xtensa-esp32s2-elf-g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Project is not inside a git repository, or git repository has no commits; will not use 'git describe' to determine PROJECT_VER.
-- Building ESP-IDF components for target esp32s2
-- Project sdkconfig file /Users/romanvg/dev/esp32s2_can/twai_network_slave/sdkconfig
-- Found PythonInterp: /Users/romanvg/.miniconda3/envs/espressif/bin/python (found version "3.9.1") 
-- Found Perl: /usr/bin/perl (found version "5.18.4") 
-- App "twai_network_slave" version: 1
-- Adding linker script /Users/romanvg/dev/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.ld
-- Adding linker script /Users/romanvg/dev/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.api.ld
-- Adding linker script /Users/romanvg/dev/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.libgcc.ld
-- Adding linker script /Users/romanvg/dev/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.newlib-funcs.ld
-- Adding linker script /Users/romanvg/dev/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.newlib-data.ld
-- Adding linker script /Users/romanvg/dev/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.spiflash.ld
-- Adding linker script /Users/romanvg/dev/esp32s2_can/twai_network_slave/build/esp-idf/esp32s2/esp32s2_out.ld
-- Adding linker script /Users/romanvg/dev/esp-idf/components/esp32s2/ld/esp32s2.project.ld.in
-- Adding linker script /Users/romanvg/dev/esp-idf/components/esp32s2/ld/esp32s2.peripherals.ld
-- Components: app_trace app_update asio bootloader bootloader_support bt cbor cmock coap console cxx driver efuse esp-tls esp32s2 esp_adc_cal esp_common esp_eth esp_event esp_gdbstub esp_hid esp_http_client esp_http_server esp_https_ota esp_https_server esp_hw_support esp_ipc esp_local_ctrl esp_netif esp_pm esp_ringbuf esp_rom esp_serial_slave_link esp_system esp_timer esp_websocket_client esp_wifi espcoredump esptool_py expat fatfs freemodbus freertos hal heap idf_test jsmn json libsodium log lwip main mbedtls mdns mqtt newlib nghttp nvs_flash openssl partition_table perfmon protobuf-c protocomm pthread sdmmc soc spi_flash spiffs tcp_transport tcpip_adapter tinyusb ulp unity vfs wear_levelling wifi_provisioning wpa_supplicant xtensa
-- Component paths: /Users/romanvg/dev/esp-idf/components/app_trace /Users/romanvg/dev/esp-idf/components/app_update /Users/romanvg/dev/esp-idf/components/asio /Users/romanvg/dev/esp-idf/components/bootloader /Users/romanvg/dev/esp-idf/components/bootloader_support /Users/romanvg/dev/esp-idf/components/bt /Users/romanvg/dev/esp-idf/components/cbor /Users/romanvg/dev/esp-idf/components/cmock /Users/romanvg/dev/esp-idf/components/coap /Users/romanvg/dev/esp-idf/components/console /Users/romanvg/dev/esp-idf/components/cxx /Users/romanvg/dev/esp-idf/components/driver /Users/romanvg/dev/esp-idf/components/efuse /Users/romanvg/dev/esp-idf/components/esp-tls /Users/romanvg/dev/esp-idf/components/esp32s2 /Users/romanvg/dev/esp-idf/components/esp_adc_cal /Users/romanvg/dev/esp-idf/components/esp_common /Users/romanvg/dev/esp-idf/components/esp_eth /Users/romanvg/dev/esp-idf/components/esp_event /Users/romanvg/dev/esp-idf/components/esp_gdbstub /Users/romanvg/dev/esp-idf/components/esp_hid /Users/romanvg/dev/esp-idf/components/esp_http_client /Users/romanvg/dev/esp-idf/components/esp_http_server /Users/romanvg/dev/esp-idf/components/esp_https_ota /Users/romanvg/dev/esp-idf/components/esp_https_server /Users/romanvg/dev/esp-idf/components/esp_hw_support /Users/romanvg/dev/esp-idf/components/esp_ipc /Users/romanvg/dev/esp-idf/components/esp_local_ctrl /Users/romanvg/dev/esp-idf/components/esp_netif /Users/romanvg/dev/esp-idf/components/esp_pm /Users/romanvg/dev/esp-idf/components/esp_ringbuf /Users/romanvg/dev/esp-idf/components/esp_rom /Users/romanvg/dev/esp-idf/components/esp_serial_slave_link /Users/romanvg/dev/esp-idf/components/esp_system /Users/romanvg/dev/esp-idf/components/esp_timer /Users/romanvg/dev/esp-idf/components/esp_websocket_client /Users/romanvg/dev/esp-idf/components/esp_wifi /Users/romanvg/dev/esp-idf/components/espcoredump /Users/romanvg/dev/esp-idf/components/esptool_py /Users/romanvg/dev/esp-idf/components/expat /Users/romanvg/dev/esp-idf/components/fatfs /Users/romanvg/dev/esp-idf/components/freemodbus /Users/romanvg/dev/esp-idf/components/freertos /Users/romanvg/dev/esp-idf/components/hal /Users/romanvg/dev/esp-idf/components/heap /Users/romanvg/dev/esp-idf/components/idf_test /Users/romanvg/dev/esp-idf/components/jsmn /Users/romanvg/dev/esp-idf/components/json /Users/romanvg/dev/esp-idf/components/libsodium /Users/romanvg/dev/esp-idf/components/log /Users/romanvg/dev/esp-idf/components/lwip /Users/romanvg/dev/esp32s2_can/twai_network_slave/main /Users/romanvg/dev/esp-idf/components/mbedtls /Users/romanvg/dev/esp-idf/components/mdns /Users/romanvg/dev/esp-idf/components/mqtt /Users/romanvg/dev/esp-idf/components/newlib /Users/romanvg/dev/esp-idf/components/nghttp /Users/romanvg/dev/esp-idf/components/nvs_flash /Users/romanvg/dev/esp-idf/components/openssl /Users/romanvg/dev/esp-idf/components/partition_table /Users/romanvg/dev/esp-idf/components/perfmon /Users/romanvg/dev/esp-idf/components/protobuf-c /Users/romanvg/dev/esp-idf/components/protocomm /Users/romanvg/dev/esp-idf/components/pthread /Users/romanvg/dev/esp-idf/components/sdmmc /Users/romanvg/dev/esp-idf/components/soc /Users/romanvg/dev/esp-idf/components/spi_flash /Users/romanvg/dev/esp-idf/components/spiffs /Users/romanvg/dev/esp-idf/components/tcp_transport /Users/romanvg/dev/esp-idf/components/tcpip_adapter /Users/romanvg/dev/esp-idf/components/tinyusb /Users/romanvg/dev/esp-idf/components/ulp /Users/romanvg/dev/esp-idf/components/unity /Users/romanvg/dev/esp-idf/components/vfs /Users/romanvg/dev/esp-idf/components/wear_levelling /Users/romanvg/dev/esp-idf/components/wifi_provisioning /Users/romanvg/dev/esp-idf/components/wpa_supplicant /Users/romanvg/dev/esp-idf/components/xtensa
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/romanvg/dev/esp32s2_can/twai_network_slave/build
Running ninja in directory /Users/romanvg/dev/esp32s2_can/twai_network_slave/build
Executing "ninja all"...
[1/13] Generating ../../partition_table/partition-table.bin
Partition table binary generated. Contents:
*******************************************************************************
# ESP-IDF Partition Table
# Name, Type, SubType, Offset, Size, Flags
nvs,data,nvs,0x9000,24K,
phy_init,data,phy,0xf000,4K,
factory,app,factory,0x10000,1M,
*******************************************************************************
[6/13] Performing configure step for 'bootloader'
-- Building ESP-IDF components for target esp32s2
-- Project sdkconfig file /Users/romanvg/dev/esp32s2_can/twai_network_slave/sdkconfig
-- Adding linker script /Users/romanvg/dev/esp-idf/components/esp32s2/ld/esp32s2.peripherals.ld
-- Adding linker script /Users/romanvg/dev/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.ld
-- Adding linker script /Users/romanvg/dev/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.api.ld
-- Adding linker script /Users/romanvg/dev/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.libgcc.ld
-- Adding linker script /Users/romanvg/dev/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.newlib-funcs.ld
-- Adding linker script /Users/romanvg/dev/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.spiflash.ld
-- Adding linker script /Users/romanvg/dev/esp-idf/components/bootloader/subproject/main/ld/esp32s2/bootloader.ld
-- Adding linker script /Users/romanvg/dev/esp-idf/components/bootloader/subproject/main/ld/esp32s2/bootloader.rom.ld
-- Components: bootloader bootloader_support efuse esp32s2 esp_common esp_hw_support esp_rom esptool_py hal log main micro-ecc partition_table soc spi_flash xtensa
-- Component paths: /Users/romanvg/dev/esp-idf/components/bootloader /Users/romanvg/dev/esp-idf/components/bootloader_support /Users/romanvg/dev/esp-idf/components/efuse /Users/romanvg/dev/esp-idf/components/esp32s2 /Users/romanvg/dev/esp-idf/components/esp_common /Users/romanvg/dev/esp-idf/components/esp_hw_support /Users/romanvg/dev/esp-idf/components/esp_rom /Users/romanvg/dev/esp-idf/components/esptool_py /Users/romanvg/dev/esp-idf/components/hal /Users/romanvg/dev/esp-idf/components/log /Users/romanvg/dev/esp-idf/components/bootloader/subproject/main /Users/romanvg/dev/esp-idf/components/bootloader/subproject/components/micro-ecc /Users/romanvg/dev/esp-idf/components/partition_table /Users/romanvg/dev/esp-idf/components/soc /Users/romanvg/dev/esp-idf/components/spi_flash /Users/romanvg/dev/esp-idf/components/xtensa
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/romanvg/dev/esp32s2_can/twai_network_slave/build/bootloader
[9/13] Performing build step for 'bootloader'
[1/1] Generating binary image from built executable
esptool.py v3.0
Generated /Users/romanvg/dev/esp32s2_can/twai_network_slave/build/bootloader/bootloader.bin
[12/12] Generating binary image from built executable
esptool.py v3.0
Generated /Users/romanvg/dev/esp32s2_can/twai_network_slave/build/twai_network_slave.bin

Project build complete. To flash, run this command:
/Users/romanvg/.miniconda3/envs/espressif/bin/python ../../esp-idf/components/esptool_py/esptool/esptool.py -p (PORT) -b 921600 --before default_reset --after hard_reset --chip esp32s2  write_flash --flash_mode dio --flash_size detect --flash_freq 80m 0x1000 build/bootloader/bootloader.bin 0x8000 build/partition_table/partition-table.bin 0x10000 build/twai_network_slave.bin
or run 'idf.py -p (PORT) flash'

[brianignacio5]

Well if you want to use a custom virtual environment you just need to set

idf.pythonBinPath or idf.pythonBinPathWin to the python binary in your anaconda venv as describe in Setup documentation. We set the python from the virtual env to avoid those activate effect that can mess with other extensions.

[brainstorm]

I'm not 100% sure that'd work, but I've moved on to "vanilla" python3 venvs to not give you guys more headaches anyway, so I'll go ahead and close this ;)