Espressif examples run with local wolfSSL (no setup!) #6018

gojimmypi · 2023-01-27T06:05:01Z

Description

Helpful for wolfSSL Espressif first-time users: This PR allows the examples to just work without needing to manually install wolfSSL as an Espressif component.

Helpful for wolfSSL Espressif developers: This PR allows live changes to be made in the code that's actually in the repo. Otherwise the executing wolfssl code would be in the $IDF_PATH components directory, with any changes needing to be copied back to the local repo to be checked in to GitHub. After doing this wonky thing as part of my build process, well... enough was enough. Allowing in-place editing is how things are supposed to work.

This PR also addresses the "exactly what version is this" question a bit more gracefully than the install-time edit of source code in PR#5955. This particular implementation is for the local wolfSSL examples only. The component install method will need a slightly different implementation.

See #6234 for a roadmap of Espressif updates.

Fixes zd# n/a

Testing

Open wolfSSL VisualGDB projects as usual, or run with ESP-IDF idf.py build flash. The examples will run without needing to install wolfssl! If wolfSSL is already installed, there will be a warning about having multiple installs, but the code should still run properly, provided the existing wolfSSL install is valid.

Checklist

added tests
updated/added doxygen
updated appropriate READMEs
Updated manual and documentation

gojimmypi · 2023-01-27T12:19:57Z

Jenkins retest this please

gojimmypi · 2023-01-29T23:07:21Z

Including a reference to espressif/esp-idf#9929 (comment) that has this valuable comment on Espressif component CMakeLists files as related to changes in this PR:

[...] component CMakeLists files are evaluated twice during the build process. First, they are evaluated early during the build in CMake script mode to extract component requirements (REQUIRES and PRIV_REQUIRES arguments of idf_component_register). Evaluation stops after encountering the call to idf_component_register() or a return(). Once the list of components has been determined component CMakeLists files are evaluated again, this time using the normal add_subdirectory call.

That's where I got the idea to wrap some items in the CMake like this:

if(NOT CMAKE_BUILD_EARLY_EXPANSION)
  # things to parse only once ...
endif()

bandi13

Over all, you seem to have a lot of duplicated code in your CMakeLists.txt. Would it make sense to have a common section of the file in a parent folder which each example references? Then you could have the unique parts in the individual examples.

Might also want to consider having a common initialization.c file for each of the main.c/wifi_connect.c files. At the very least have consistency in calling the file main.c.

bandi13 · 2023-01-31T14:42:15Z

IDE/Espressif/ESP-IDF/examples/wolfssl_server/main/wifi_connect.c

+
+#if defined(WOLFSSL_MULTI_INSTALL_WARNING)
+    ESP_LOGI(TAG, "");
+    ESP_LOGI(TAG, "WARNING: Multiple wolfSSL installs found.");


This seems like something that should be a compile time warning. Not runtime.

Well, it's both: as the make/build process is rather long and the build warning could be missed.

It would be good to include the app as well for higher visibility.

bandi13 · 2023-01-31T14:45:24Z

IDE/Espressif/ESP-IDF/examples/wolfssl_server/main/CMakeLists.txt

+    # wolfSSL found in both ESP-IDF and local project - needs to be resolved by user
+    #
+    message(STATUS "")
+    message(STATUS "WARNING: Found components/wolfssl in both local project and IDF_PATH")


Maybe consider making this an error and not just a warning. Bad/unexpected things could happen because of an unreliable location of the component source files.

I had considered making it a failure. But I wanted to to be a "it just works" feature. Sure, the user may have an different version installed, but do you think that should really make it a hard fail? This method we are pointing out which is being used, with a warning in the app (noted above).

It might however, be a good idea in a future revision to determine the versions of each and display that information as well.

I've reconsidered this and yes: I'll submit a change to make this a hard fail error. The risks of mixing code from each of the (potentially different version) component directories if the user ignores the warning seems ripe for support headaches.

bandi13 · 2023-01-31T14:48:44Z

IDE/Espressif/ESP-IDF/examples/wolfssl_test/components/wolfssl/CMakeLists.txt

+                message(STATUS "")
+                message(STATUS "WARNING: wolfSSL not found.")
+                message(STATUS "")
+            else()


Remove unnecessary 'else' block

Good catch! Actually I was missing a message, so the else() stayed and I added a message regarding path.

gojimmypi · 2023-01-31T18:35:03Z

duplicated code in your CMakeLists.txt. Would it make sense to have a common section of the file in a parent folder which each example references?

I assume you mean across the different projects? Yes. This is as intended; I realize that there could be a more efficient way for this group of examples. Instead, this design pattern is for portability: just take the CMakefiles.txt, plunk it into a project ./components/wolfssl/ and assign the appropriate WOLFSSL_ROOT value. Having some sort of CMakefile include reference would reduce that flexibility and result in something much more fragile.

gojimmypi · 2023-02-01T04:59:17Z

duplicated code in your CMakeLists.txt

In the latest update, I added a CMake function that is easier to understand and use.

gojimmypi · 2023-02-05T23:55:43Z

Jenkins retest this please

gojimmypi · 2023-02-06T19:49:57Z

Jenkins retest this please

…essif_No_Install

wolfSSL-Bot · 2023-02-09T20:58:20Z

Can one of the admins verify this patch?

dgarske

On Mac:

./IDE/Espressif/ESP-IDF/compileAllExamples.sh
Found IDF_PATH = /Users/davidgarske/GitHub/esp-idf
~/GitHub/wolfssl/IDE/Espressif/ESP-IDF/examples/wolfssl_benchmark ~/GitHub/wolfssl
Traceback (most recent call last):
  File "/Users/davidgarske/GitHub/esp-idf/tools/idf.py", line 45, in <module>
    from idf_py_actions.tools import (executable_exists, idf_version, merge_action_lists, realpath)  # noqa: E402
  File "/Users/davidgarske/GitHub/esp-idf/tools/idf_py_actions/tools.py", line 1, in <module>
    import click
ImportError: No module named click
~/GitHub/wolfssl
Failed in benchmark

Both python and python3 have "click" installed...

% pip install click
Requirement already satisfied: click in /usr/local/lib/python3.10/site-packages (8.1.3)
davidgarske@Davids-MBP wolfssl % pip3 install click
Requirement already satisfied: click in /usr/local/lib/python3.10/site-packages (8.1.3)

gojimmypi · 2023-02-13T20:13:19Z

@dgarske Thanks for taking this for a test drive. Can you please confirm you have a standard ESP-IDF release install? It appears from the message that you may have a special (GitHub clone? modified? non-release?) version in your /Users/davidgarske/GitHub/esp-idf directory. That's typically not a default install location.

Could you please try from the default Espressif release install and let me know the output? For example:

# switch to your installed test of PR6018
cd ~/
git clone https://github.com/gojimmypi/wolfssl.git wolfssl_PR6018b_test
cd ~/wolfssl_PR6018b_test/IDE/Espressif/ESP-IDF 
git checkout Espressif_No_Install

# setup ESP-IDF
. ~/esp/esp-idf/export.sh

# build all the examples
./compileAllExamples.sh

I'm also curious as to exactly which version you are attempting. Could you please let me know the result of idf.py --version

I've tested this with both ESP-IDF v4.4 and v5.0 on WSL and a "real" Ubuntu VM, but I don't have a Mac to test your exact setup.

edit: Also consider building individual projects:

cd ~/wolfssl_PR6018b_test/IDE/Espressif/ESP-IDF/examples/wolfssl_test
idf.py build

dgarske · 2023-02-13T21:02:11Z

That worked. I needed to checkout the normal esp-idf master and run the ./install.sh.

gojimmypi · 2023-02-13T22:11:26Z

Note there's a new hard fail for wolfSSL being installed in both the ESP-IDF components directory and this new local component makefile. After a discussion with @bandi13 we decided it would be an unnecessary challenge for support in determining which source was being used, and perhaps even a non-zero chance of cross-contamination between them when each is a different version.

Fixed

Espressif examples run with local wolfSSL (no setup!)

ed2a6f0

gojimmypi requested a review from dgarske January 27, 2023 06:05

gojimmypi self-assigned this Jan 27, 2023

gojimmypi added the enhancement label Jan 27, 2023

gojimmypi requested a review from bandi13 January 27, 2023 06:05

include.am Espressif local no-setup component files

59880d5

dgarske self-assigned this Jan 27, 2023

This was referenced Jan 29, 2023

Target-specific autogen? Espressif in particular. #5883

Closed

[Bug]: esp32_sha.c:511:26: error: lvalue required as left operand of assignment #6026

Closed

Support for Espressif ESP-IDF v5 #5909

Closed

bandi13 previously requested changes Jan 31, 2023

View reviewed changes

dgarske removed their assignment Jan 31, 2023

This was referenced Jan 31, 2023

append git version info during Espressif install #5955

Closed

New Espressif Component Manager Integration Needed wolfSSL/wolfssh#428

Closed

New Espressif Component Manager Integration Needed #5319

Closed

cleanup Espressif Example CMakeLists.txt, use function

50cd511

gojimmypi requested a review from bandi13 February 1, 2023 05:10

gojimmypi added 3 commits February 1, 2023 19:46

multiple wolfSSL installs is now a fatal Espressif build error

6c9dbcc

Examples no longer need setup

2fd747d

CompileAll builds local examples, not IDF_PATH ones

c68c039

gojimmypi mentioned this pull request Feb 4, 2023

Espressif idf refuses to clean shared VisualGDB build directory #6057

Closed

gojimmypi added 5 commits February 5, 2023 12:42

Espressif compileAllExamples both local & ESP-IDF components

0e013ee

add wolfssl_test_idf test project

f657e8f

move VisualGDB projects to subdirectories

2dde81b

move VisualGDB wolfssl_server to subdirectory

3bd0531

update include.am for moved VisualGDB project files

1c7a513

Merge branch 'master' of https://github.com/wolfSSL/wolfssl into Espr…

2911b6d

…essif_No_Install

gojimmypi marked this pull request as draft February 6, 2023 20:17

Merge branch 'master' of https://github.com/wolfSSL/wolfssl into Espr…

5c876b9

…essif_No_Install

gojimmypi marked this pull request as ready for review February 7, 2023 13:20

gojimmypi assigned dgarske Feb 10, 2023

dgarske requested changes Feb 13, 2023

View reviewed changes

dgarske removed their assignment Feb 13, 2023

gojimmypi requested a review from dgarske February 13, 2023 20:13

gojimmypi assigned dgarske Feb 13, 2023

dgarske approved these changes Feb 13, 2023

View reviewed changes

dgarske merged commit ed79545 into wolfSSL:master Feb 13, 2023

gojimmypi mentioned this pull request Feb 14, 2023

Espressif idf refuses to clean shared VisualGDB build directory wolfSSL/wolfssl-examples#369

Open

gojimmypi mentioned this pull request Feb 22, 2023

Feature Request: local project user_settings.h for no-setup Espressif #6118

Closed

dpalffy mentioned this pull request Apr 3, 2023

Software-only support for recent esp32 targets #6256

Open

4 tasks

This was referenced May 3, 2023

Update Espressif examples to use new "no setup" CMakeFiles.txt wolfSSL/wolfssl-examples#385

Open

Update Espressif SSH examples to use new "no setup" CMakeFiles.txt wolfSSL/wolfssh#514

Closed

gojimmypi mentioned this pull request Dec 31, 2023

ESP32 Espressif Improvements - Roadmap Summary #6234

Open

72 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Espressif examples run with local wolfSSL (no setup!) #6018

Espressif examples run with local wolfSSL (no setup!) #6018

gojimmypi commented Jan 27, 2023 •

edited

Loading

gojimmypi commented Jan 27, 2023

gojimmypi commented Jan 29, 2023

bandi13 left a comment

bandi13 Jan 31, 2023

gojimmypi Jan 31, 2023

bandi13 Jan 31, 2023

gojimmypi Jan 31, 2023

gojimmypi Feb 1, 2023

bandi13 Jan 31, 2023

gojimmypi Feb 1, 2023

gojimmypi commented Jan 31, 2023

gojimmypi commented Feb 1, 2023

gojimmypi commented Feb 5, 2023

gojimmypi commented Feb 6, 2023

wolfSSL-Bot commented Feb 9, 2023

dgarske left a comment

gojimmypi commented Feb 13, 2023 •

edited

Loading

dgarske commented Feb 13, 2023

gojimmypi commented Feb 13, 2023

Espressif examples run with local wolfSSL (no setup!) #6018

Espressif examples run with local wolfSSL (no setup!) #6018

Conversation

gojimmypi commented Jan 27, 2023 • edited Loading

Description

Testing

Checklist

gojimmypi commented Jan 27, 2023

gojimmypi commented Jan 29, 2023

bandi13 left a comment

Choose a reason for hiding this comment

bandi13 Jan 31, 2023

Choose a reason for hiding this comment

gojimmypi Jan 31, 2023

Choose a reason for hiding this comment

bandi13 Jan 31, 2023

Choose a reason for hiding this comment

gojimmypi Jan 31, 2023

Choose a reason for hiding this comment

gojimmypi Feb 1, 2023

Choose a reason for hiding this comment

bandi13 Jan 31, 2023

Choose a reason for hiding this comment

gojimmypi Feb 1, 2023

Choose a reason for hiding this comment

gojimmypi commented Jan 31, 2023

gojimmypi commented Feb 1, 2023

gojimmypi commented Feb 5, 2023

gojimmypi commented Feb 6, 2023

wolfSSL-Bot commented Feb 9, 2023

dgarske left a comment

Choose a reason for hiding this comment

gojimmypi commented Feb 13, 2023 • edited Loading

dgarske commented Feb 13, 2023

gojimmypi commented Feb 13, 2023

gojimmypi commented Jan 27, 2023 •

edited

Loading

gojimmypi commented Feb 13, 2023 •

edited

Loading