-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] ping failed and rtps test failed. #12
Comments
Hi, do I understand correctly that you are not able to ping the STM? If you are unable to ping the STM then something with your network or lwip config on the STM side is wrong. Can run your project without starting embeddedRTPS (comment out embeddedRTPS-STM32/stm32/Src/main.cpp Line 535 in f21d3fa
|
@akampmann thanks for your quick reply!
Yes, ping 192.168.1.103(stm32) outputs unreachable
Yes, I can ping the stm32 without starting embeddedRTPS, is there something wrong with the prio setting of threads? I've print the task info above. This behavior happens also in microROS-ertps demo in https://github.com/micro-ROS/micro_ros_stm32cubemx_utils As for this repo, I've changed some code, but I believe the changing is about
You can check with this ZhenshengLee@3b983a3 Thanks. |
I will have a look and come back to you. I haven't been running embeddedRTPS on the STM32 code for a while. I've been mainly developing for the Xilinx UltraScale+ Cortex R5. |
@ZhenshengLee I've created a new branch that tracks the latest embeddedRTPS, CubeIDE, fw v1.16.2, FastDDS v2.3.2 (probably also works with the latest version): STM32 is now able to talk again to the Linux side, could you give it a try on your side? I seem to have sporadic issues with pinging the STM, but that occurs regardless of including embeddedRTPS, please let me know if you observe the same behavior. |
Hi @akampmann Great Library! Extremely cool to see the RTPS-DDS running on a Micro-controller I have been trying to get the example project working (the new branch called /feature/track_latest) and i have had some issues trying to get the example working. I have used the branch as is and set my Unix machine Networking Information to: Address: 192.168.1.104 I have also been using the latest version of FastDDS I have connected the STM32F7 directly to my computer and noticed that it was hanging when it was trying to match the publisher and subscriber. I did some debugging and found that it was failing to join IGMP multicast group (The printf verbose debugging is extremely useful). Inside the STM32F7 in the stm32f7xx_hal_eth.c file i had to change
to After changing this i was able to successfully join the IGMP group however i now get the error "ParticipantProxyData deserializtaion failed" when the handleSPDPPackage function is called. I haven't found the cause of the "ParticipantProxyData deserializtaion failed" but am continuing to debug. If you have any comments on potential causes or tests to help narrow the problem down it would be great to hear them! Cheers Jay |
@akampmann Thanks for your quick reply! I am on my vacation now, I'll give it a test in 10 days. |
@jayvanmali thank you for trying out the new branch and providing feedback. I'm curious why the changes to the MAC configuration were not necessary on my side. Regarding the deserialization failure: Could you please try FastDDS v2.3.2? That is the exact version that I tested on my side. There are probably changes to the SPDP messages in newer FastDDS version which have not been incorporated in embeddedRTPS yet. Such changes were also necessary in the past. I will try to find time to look into this. Meanwhile, could you try 2.3.2 on your side and come back to me? |
Hi @akampmann, Thanks for your prompt reply. Im currently away at the moment, but will test it with the FastDDS v2.3.2 version when i'm back (with in a day or so). In regards to the MAC configuration changes, Once i get the example working, I will change it back to the same settings are provided in the example and see if it works. This should help understand the MAC configuration change. Cheers Jay |
Hi @akampmann So i have installed FastDDS v2.3.4 (I couldn't find the files for v2.3.2) and i still get the error:
Just to be safe, i cloned the files from GitHub (And swapped to the /feature/track_latest branch) and did the following (Determined by trial and error as well as the github read-me):
Here is a copy of the log from terminal: debuglog.txt If you have any comments on potential causes or tests to help narrow the problem down it would be great to hear them! I'm still making my way through debugging and will let you know when i manage to get it working! Cheers Jay |
this has most likely to do with the recent changes introduced that avoids adding remote locators that are not on the same subnet as embeddedRTPS. Could you please try to change this line to: to
The function returns false to indicate that the locator is not added, but that return is incorrectly causes the deserialization to stop. It worked on my side because since the first remote locator transmitted by remote participants was on the local network.... Do you mind confirming that this fixes the issue on your side? I will then create a fix on the master branch shortly. |
I've updated https://github.com/embedded-software-laboratory/embeddedRTPS-STM32/tree/feature/track_latest to incorporate the fix. It works for me on the Cortex R5 now. Would you mind giving it a try on your side? |
Hi @akampmann Great! ill give it a test a bit later this afternoon, Thanks for looking into it so quickly! |
The code has been altered to https://github.com/ZhenshengLee/embeddedRTPS-STM32/tree/outdoor/feature/track_latest/ to get a better debug experience.
issuethe serial output behavior(the callback of rtpstest thread) is abnormal expected
actuala lot of screenshotsee the timestamps from the ternimal commentis the serial or the callback work abnoraml?
EDIT: the test happens in 2022-0509-08:41 of your timezone, after your branch being updated for @jayvanmali 's issue zs@zs-3630:~$ TZ='Europe/Berlin' date
2022年 05月 09日 星期一 08:41:15 CEST |
issueoh, screenshot |
after a 2-hour test of ping, I think there's no issue of the ping function of the stm here is my board and toolchain info:
screenshot |
I think your new branch https://github.com/embedded-software-laboratory/embeddedRTPS-STM32/tree/feature/track_latest of work can be merged as the basic function(ping and rtpstest) has been recovered. So, after the feature branch being merged into master
|
@ZhenshengLee Can you add a newline character at the end of the output? Maybe the output buffer is only cleared and transmitted once full or after occurrence of a new line:
@ZhenshengLee @jayvanmali This should be fixed on the latest master.
@ZhenshengLee Could you please run with gdb and provide the output of backtrace once it runs into the segmentation fault? |
I have tried the new commit and the error for Ive attached the RTPS debug log: RTPS_DEBUG.txt Im also curious to know why i have to adjust the stm32f7xx_eth.c and the ethernetif file to successfully join the IGMP group. @ZhenshengLee did you have this issue when running this example? |
@jayvanmali Are you running this by any chance in a virtual machine? Could you provide a wireshark recording of the entire interaction? |
Nope, during my test, I did not change anything in eth setting of the stm32. |
Hi @akampmann I'm running this on a Linux laptop with a HP USB-C to RJ45 Adaptor as the laptop does not have a Ethernet Port. I have attached the RTPS Log from the STM32F7 and the Wireshark Recording: RTPS Log: STM32_Log.txt After it appears to freeze, the pinging does not work anymore, I am doing some debugging to workout where its locking up but i cant immediately see anything failing. Be happy to try any suggestions you recommend. Cheers Jay |
That is interesting, Are you able to confirm if your stm32f7xx_eth has promiscuous mode enabled?
Can you also check your ethernetif and see if you have the IGMP flag set: if i don't enable these i cant seem to join and IGMP group. Much Appreciated Jay |
Something is off with the IP address configuration. The subnet assumed by embeddedRTPS is wrong (it sends messages to 192.168.42.44 (thus it assumes that this locator is on its own subnet)), but IP packets are actually sent from 192.168.1.103. Can you confirm that the IP addresses here: embeddedRTPS-STM32/stm32/LWIP/App/lwip.c Line 69 in 6f9e84a
And here:
match? Currently configuring the IP address is a bit messy, as it has to be changed in two places. That is mainly because there does not seem to be a consistent way of configuring the IP address of the lwIP interface on different platforms (its different on the Cortex R5). I will have to add an assert at some point to warn of this misconfiguration during embeddedRTPS init. |
This is just an FYI. I've confirmed feature/track_latest branch could work well in my environment as is (just changing the IP address). I have the utmost respect for @akampmann 's work!! |
@takasehideki thank you so much! @jayvanmali did you have a chance to look at the IP config? |
Hi @akampmann Apologies for the delayed response. I have been away for the past week and will back end of this week, Ill confirm the branch then and let you know my findings. Cheers |
Hi @jayvanmali did you have a chance to try the fix? |
Hi @akampmann, apologies for the delay, An Illness left me out of action for a while. I'll test it within the next day and report back, I'll attach the trace log and Wireshark log as well Cheers Jay |
Hi @akampmann I've started trying the fixed, but have had network issues on the F7 which i have narrowed down to hardware issues, I'll continue testing the fix when i get access to a second F7 (within a day or so). Cheers |
yes, you can check in https://github.com/ZhenshengLee/ros2_mcu/blob/0cdedf3f9f07599bb58fbd42e61ac5204fdfddd0/libs/custom_stm32f767zi_ertps/LWIP/Target/ethernetif.c#L241 |
hey @jayvanmali, did you have a chance to try with a new F7? |
hey @akampmann, Apologies for the radio silence, I have been away. Sadly due to supply issues, i have not been able to get an F7 but i do have a H723. I'm going to retry and get this working on the H7. I'm fully back on deck so i have more time and ill keep you updated on the progress, ill also use the newest version of embeddedRTPS. I have done a quick integration of it but it seems to fail joining the IGMP group so i need to check the MAC filtering. Cheers for everything so far! |
Hey @akampmann I have managed to test the new code on a STM32H723, Sorry for the late response. I managed to fix the IP Config I still seem to have a problem with the matching of the Publishers and Subscribers. I have attached the Wireshark Log and the Embedded RTPS DDS Log. If you have some thoughts, Happy to try any solution. Cheers Jay |
Hi, the problem is simple, your interface IP address does not match the embeddedRTPS config. You are sending data from 192.168.1.47, but the discovery advertises 192.168.1.103 as the IP address of your STM. You need to adapt this parameter to 192.168.1.47
Unfortunately, you are not the first one to encounter this issue. I will try to include a warning or assert in the embeddedRTPS code to catch this problem. |
Hi @akampmann Good catch! I fixed that change, but i still seem to have the STM32 code remain waiting for the Publisher and Subscribers to match. Any thoughts on what might be causing that? Cheers Jay |
Can you provide a Wireshark recording with the adapted config? Are you able to ping the STM? |
Hey @akampmann I can successfully ping the STM32H7 on 192.168.1.47. Here are the new New Logs Here is the RTPS config file: What factors would stop the publishers and subscribers from matching? Cheers Jay |
Hi @jayvanmali your machine does not respond to the STM32. Here are some thoughts:
Best regards |
Hey @akampmann I think i have manged to get something working, It appears to be a issue with the version of FastDDS used in my ROS distribution (Locked to 2.6) and a second issue with the Firewall on my laptop On a downgrade to 2.3 and just using FastDDS directly, it appears to be working. Do you know what changes are potentially required to enable it working with FastDDS 2.6? Happy to look into it if you have any comments. Best Regards Jaynesh |
hey @jayvanmali in the past, the main issues were increased discovery message sizes, due to additional fields. I will try to replicate this problem. Have you tried turning verbosity on? Best regards |
Hey @akampmann I've turned on the debugging and am slowly going through the different versions of FastDDS and seeing when and if it fails and finding out why it does at those points, I will keep you updated with what i find. One thing i have noticed is that when i try to add another participant in the same domain, i seem to keep getting a hard-fault even though my MAX_NUM_PARTICIPANTS in the config file is 2. I have made sure my FreeRTOS threads have a stack size of 5000. After stepping though the debugger it seems to fail at the "addBuiltInEndpoints". Do you have any idea of why this could be the case? or could i be missing a config setting to have multiple publishers and subscribers? Cheers for all the help Best Regards Jay |
Hey @jayvanmali I've finalized my thesis and now had some time to look into this. I have tested embeddedRTPS both with FastDDS v2.6.0 and v2.8.0 - It worked with both versions on my side without requiring modifications to embeddedRTPS Just to be clear: The issue on your side occurs in conjunction with ROS on top of embeddedRTPS, right? Can you confirm that FastDDS 2.6.0 and embeddedRTPS (without ROS2) works on your side? From my experience of using embeddedRTPS in a different context - these kind of issues have almost always been related to insufficient stack sizes of the reader workers. Depending on what you are doing in the context of the callback... Can you try to use the following code snippet to print the stack consumption? Make sure to run it periodically or at least close to where you see your application crashing.
Most likely you will have to increase (at least) the following stack sizes
|
@ZhenshengLee the core dump is also fixed on master, I was not using FastDDS correctly. |
Thanks for all authors! But I just cannot reproduce the behavior of the ertps-stm32.
Here is the issue.
Env
Steps
change setting according to https://github.com/embedded-software-laboratory/embeddedRTPS-STM32/blob/master/readme.md
stm32/rtps/include/rtps/config.h
stm32/rtps/thirdparty/lwip/lwipcfg.h
stm32/rtps/include/rtps/config.h
stm32/Src/lwip.c
Expected
../embedded_rtps_test
show something/Actuall
../embedded_rtps_test
waitingCreating RTPS Participant... Creating Participant... Created Participant succesfully. Creating RTPS reader on topic:TOLINUX Creating FastRTPS Writer on topic TOSTM Successfully created writer. Waiting for reader match with STM32...
192.168.1.103
disable the wifi makes no different of the result.
wireshark capturing no packets from
192.168.1.103
I added the tasklist function to show the thread report below
The text was updated successfully, but these errors were encountered: