-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve performance of the control loop #7
Comments
I'm also trying to figure out why I'm only getting 5 Hz for the
It seems like the Reference: |
The find should be really negligible here. I think this comes more from how we broadcast commands and wait for replies. Also, do the dynamixel_workbench code query the position of each servos at each time step? This takes a significant time (versus sending commands and that's it). |
I believe so. Although, they're also reading data for each servo (and not using "sync read" or "bulk read"), so I'm not sure where the difference comes from 😕 That being said, I haven't actually used dynamixel_workbench myself so I'll need to test it out on my servos to do a comparison. |
One difference between DynamixelSDK and Libdynamixel is that they use a non-blocking call to read(). We were already considering this option which shall be tested (hopefully soon). Also, They have slightly different settings for the serial interface, and there is a latency_timer thing. For further reading, their reading method (for protocol 1) is here.. |
This problem is under investigation. I plan on going back to it from today on. If you want to help on this investigation, it seems that most of the time is waisted in |
👍 Do you think it's not necessary to use |
I think that it should help but I am not sure that it is the root cause, since it seems that robotis themselves do not use it. |
I found a few, very interesting, lines in DynamixelSQK. They point to It appears that by putting it to 1 (ms), we were able to run the control loop at 100 Hz with 5 actuators. @naoki-mizuno could you confirm this observation ? Next I'll see how we can use commands to read and write for all joints at once. This is expected to further improve the loop frequency. |
@dogoepp I gotta say, you just made my day 😆 I set the latency timer to 1, and I was able to get the joint states at 50 Hz with 7 servos. I'm not sure how I overlooked that parameter when reading through their code, but I'm pretty happy with how smooth our manipulator works now! |
Great ! I hope that the next step will make it even better. |
Hi sorry for digging this thread up. I just stumpled on it randomly |
Hi @SammyRamone , |
In my case MX-64 and MX-106. If you update the firmware you can use up to 4.5Mbaud. In reality 4.5Mbaud doesn't seem to work, but 4Mbaud does. |
Sounds great. I think indeed that using sync read and writes could help a lot. If I recall correctly, either of these commands is only available in the version 2 of the protocol however. Good to know that implementing this would allow for much greater loop frequencies. As for the firmware, it might be that some older versions of MX-28 cannot be upgraded. Out of curiosity, could I ask why you have implemented your own hardware interface? Was the barrier of entry to use this one too high? |
Actually you can also use sync reads in the protocol version 1. Out of some reason its not documented, but it works. We used to do it before we switched to version 2. If you want to do it I can dig out our old code. Yes thats possible. I remember that we had some problems with updating MX-28 servos but we didn't bother since we stopped using them. Actually I implemented most of this over a year ago. I think at that point your software was not finished. I just had a look at it today because I thought about switching to yours since it doesn't make sense to have thousands of implementations for the same thing. But currently I'm missing the sync read and write functions. Btw: Did you set the return delay time in the servos to 0? This is also a common source for low frequency. I don't really know why robotis did set the default value to something else than 0. Second btw: we just developed a multi-bus controller board solution which enables us to read/write 20 servos with 1.3KHz, if you need even higher frequencies |
I would like to help, but I do not have access to Dynamixels anymore and am rather busy. If @PedroDesRobots / @dalinel and/or you are willing to go this way, I would be happy to give a hand, at least to find the parts in the code that need modification. I was not aware of the return delay time. It should be checked. About your board, how is it connected to the computer? How do you handle the multiple buses? |
Firstly, I have set the return delay time in the servos to 0. It was the default value 250 ms... as expected it increases the transmission speed. I was plotting time delay in my controller loop (update function) and that divided by two the time. Concerning updating MX-28 servos I will try it soon. And I will take a look at sync reads and how to add it in our code. @SammyRamone could you tell us more about your multi-bus controller? Because we are really interesting to increase our frequencies to be able to produce better controllers in velocity mode. |
I recommend to always set the return delay time to 0 automatically in all servos when starting up your software. Otherwise you will change a servo or update firmware at some point and wonder why your update rate dropped again. Trust me, I've been through this :D Regarding updates of the MX-28: I think the older versions had a chip with less memory and the new firmware doesn't fit on it. Therefore they don't work. But I'm not totally sure about this. Regarding the multi-bus: We just took an multi channel FTDI chip (similar to the robotis U2D2 but with 4 channels). Currently we just let our software run 4 times on the different bus systems for tests. But it should be possible to have 4 hardware interfaces where the controllers connect to, but I didn't try it yet. |
I have upgraded the MX-28 firmware (v36 to v39) under the software RoboPlus. It was impossible to do it under R+ Manager... I don't know why but R+ never detect my servos but I suspect to use protocol 2 only. Now I can increase the baudrate at 3 Mbauds (it didn't propose 4MBauds in the soft...) BUT when I'm trying to read address 5, which is the address number in control table for baudrate, it returns 1 (1Mbaud in protocol 1 or 57600 in protocol 2). In the end, the servo responds only at 3Mbauds expected result but only under protocol 1. I set the return delay time at 0 but it responds only for a --scan-timeout of 0,05. While with MX-28 v36 set at 1 Mbaud we are able to reach a scan-timeout of 0,001. @SammyRamone could tell me which soft do you use to update servos (RoboPlus or R+) and how do you manage to communicate with protocol 2 ? Btw, with our MX-28 servo v36 set at 1 Mbaud we are able to reach 1000Hz in our control loop. when I'm trying to communicate with 18 servos, the best result is around 20Hz.... and normally we should be able to reach 50Hz easily (~1000Hz / 18 servos ) but I guess it's not linear. Of course, our solution to use multi-channel FTDI chip is the best way to carry on to increase speed. Which kind of multi-channel FTDI do you use? If you can send me the paper it will be pleasant. |
So, 1khz with a single Mx-28, but 20Hz with 18 Mx-28. Right? Which usb2dynamixel did you use? we have three options. We could also simply use 4 USB adapters with a good USB hub. Also, @SammyRamone might be using the 4-wires servos. |
@jbmouret four wires or three wires are the same in term of communication : half-duplex. @PedroDesRobots could you clarify your second paragraph? It is hard to figure what are your observations, under which circumstances. In addition, It may be simpler to start by moving to sync read and write before going the way of multiple hardware controllers. This is no modification to existing hardware and has the potential of an immediate improvement, for free, for all users of this software. |
Hi, sorry for replying so late. There were just too many deadlines. Regarding the MX-28: I'm not totally sure about these, since we stopped using them. You should be able to differentiate the versions by the chip that is installed in them. But again, I'm not totally sure about the 28s. Regarding your 20Hz : I would guess one of them still has a return delay time. Best way to investigate this is using a logic analyser. I can recommend the one from Saleae. That way you can actually see what is happening on the bus. @jbmouret I used RS-485 and TTL both (4 and 3 wires). There is no real difference in them, except that RS-485 works better on long distance transmissions and on bad cables. The paper is unfortunately still in a state where I can't send it around. But I can sum up the most important points. When the paper is ready, I will be happy to share it with you. |
I have some problem getting high loop speeds. I am running 12 x MX-28R in one setup with U2D2 and the servos are updated to protocol 2.0 and running at 4 MBPS. I get around 50hz. The servos are Daisy chained in Groups of 3. My second setup will be 8 x MX-106R and 4 x MX-64R. Also Daisy chained in Groups of 3. But havent tried the loop speed on this setup yet…. I have set the Any suggestions ? :) |
Hi The specialists are @Aneoshun and @PedroDesRobots (but he is on vacation this week). We definitely get much higher frequencies with MX-28. We have limited access to the lab right now (covid-19 restrictions), but we can have a look at the end of the week, maybe. |
thanks @jbmouret I did some troubleshooting last night an it seems that I forgot to set I read that @SammyRamone suggest to set btw: First time on the legs |
Nice video and congratulations ! It's always nice to get to the point where the robot actually does something. I guess that @jbmouret or @PedroDesRobots will know the exact loop frequency we can get for our robots (those with 18 servos). I think that it could run at 100 Hz. If I recall correctly the driver would always make us wait the duration of I would then advise to put this setting to a low value. I think that we used as low as 0. |
The U2D2 should be fast enough to give you a decent update rate with only 12 servos.
Generally speaking, when having such problems, you should investigate what is actually running on your bus, to see what takes which amount of time. The easiest way to do this, is to use a logic analyser. I'm using one from Saleae and it works very well. |
Thanks @SammyRamone for the detail respons. Will try out some of your suggestions when I get home from work. Some commments.
best regards |
I don't have experience with the libdynamixel, but generally a bulk_write is still slower as a sync_write due to having a larger package length. Still, it is much better than using single writes. I'm using forks of the DynamixelSDK and Dynamixel Workbench from Robotis. I implemented reading multiple registers with a single sync read there. You can find it here: |
Nice :) |
Yeah didnt check :( just ran the dynamixel controller from here without modifying anything. btw : if I run your suggestion for latency_timer I cant find any dynamixels. If I stick with latency_timer = 1 it works fine. I tried to brake out your bitbots_ros_control without all the extras. But still havent got it to work yet :( I would rather use this lib since my robot is working already with it. But the hz is still a problem. |
The code looks like they are doing a single sync read for all 3 values. Seems like they improved their software over the last years. Then the 330Hz are also very close to what my formula would predict. Looks like the MX-28 are behaving similar to the MX-64/106. Hm, thats strange. I already used it on multiple computers and it worked. But I didn't use the U2D2 but my own controller board, maybe there is a difference. Maybe it also has to do something with different kernel versions or something. Yeah our code has a lot of specific extras. I would also think, that the standard dynamixel_workbench would be fine for you. |
@SammyRamone robot is up and running with workbench. I am able to hit 930hz on dynamixel/joint_states with a single U2D2. 😇 But now the robot is shaking a bit instead 🤔 I just tought you would like that info. |
So for the latency error, I have no idea. Hm, maybe have a look on the data. It could happen that this library "claims" to have 930Hz while not actually giving the robot this frequency. I can recommend to move the servo and use PlotJuggler, with points activated in the plot, to have a look at the current position. If all values are different and you have a smooth curve its great. If you have some doubled values, the library is not really reading the servo at this speed. If you have longer periods of the same value, you have package loss and timeouts. Thanks for the info, its interesting for me to know :) |
yeah 930Hz sounded higher than possible. The 320Hz the topic monitor shows is more realistic. Your video looks a bit more like a problem with the PID controller (oscillation). But a plot might give more insight. |
Yeah maybe. But If I change the loop frequency of the controller rqt give me high frequency again. Like 800-1000hz. What plot would be helpful to check? Just check the joint position plot and see how smooth the curve is? Yeah maybe it is something with the PID. But if I start the dynamixel_workbench without trajectory controller i don’t experience any oscillation. The joint are stiff. If I start the trajectory controller the oscillation starts and the joint are not as stiff as before. I tried to adjust the PID to original values which worked in the ResiBots hardware interface but without any major success. Here is the latest test |
Just plot the current position of a joint (from joint_states message) while you move it. If you get multiple times the same value after each other, the servo is not actually read as often as the message is published. Maybe it has something to do with the trajectory controller. I don't have much experience with it. |
Hi @SammyRamone and @raess1, Thank you for your interest in your library and its performance. It's clearly not an easy task to compare all these things together. For a short summary, the performance of our library depends on :
Note 1: the performance it's clearly not linear which is a counter-intuitive result. |
@PedroDesRobots This is with a 1Mbs baudrate? |
Yes, I forgot to mention that. So with MX-28 updated with protocol 2 and 4 M Bauds, the performance should be higher. |
Thanks for the slide @PedroDesRobots. As I said I am running 4MBaud with latest firmware on P2. The frequency seems a lot higher on the workbench controller. |
I was using libdynamixel branch dev adn dynamixel_control_hw branch MX. You should also set the latency timer to 0, but I guess you already did that. Let me know if you got some trouble installing these branches. |
@PedroDesRobots We should plan to merge these branches, right? |
@jbmouret yes completely for the libdynamixel. Concerning dynamixel_control_hw with the templated function is hard to be generic, we have to specify the actuator type. |
I can give a hand here, at the very least through review of the code.
Otherwise, on well-defined, finite-time tasks.
Le lun. 6 juil. 2020 à 15:03, PedroDesRobots <[email protected]> a
écrit :
… @jbmouret <https://github.com/jbmouret> yes completely for the
libdynamixel.
Concerning dynamixel_control_hw with the templated function is hard to be
generic, we have to specify the actuator type.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#7 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADE2O5FHIARPLC7TUB77VITR2HDRBANCNFSM4B6O2NFQ>
.
|
@PedroDesRobots Let me make a clean install with does and check frequency for a single Mx. I be back 😎 |
I am trying to catkin_make my workspace after install the libdynamixel. (I followed the guide for
|
Just as a side-note, @raess1, could you please wrap code or raw text, like these errors, in triple backticks (this character : `) on blank lines ? It makes it more readable, as it uses a fixed-width font. |
roget that @dogoepp fixed code above :) |
Hi @raess1, Did you install the right branch? libdynamixel branch dev and dynamixel_control_hw branch MX? |
@PedroDesRobots yep I did git checkout on does branches! |
And did you clean properly your old install on libdynamixel, then reinstall it? This error : |
@PedroDesRobots I git cloned libdynamixel in resibot_ws/src Did these steps Then added path for new install in the CMakelist.txt
And removed lines 24-38 CMakelist.txt Since they where not working for me.
|
Hello guys, I just read through your discussion. I am writing my masterthesis on the control a two-link manipulator using two Dynamixel XM340-W210 Servos. I was wondering how to further reduce the latency when reading out the position and velocity values. Regards, |
The default frequency for this control loop is 10 Hz for us. Let's set
cycle_time_error_threshold
to 0.1, i.e. the maximum allowed duration of an iteration to ensure 10 Hz iteration frequency. I observe on Positronic that the threshold is regularly exceeded (see bellow). Admittedly, the excess time is very little, but it shows that we are already close to the limit.One possible axis of improvement is the find operations on _dynamixel_corrections in each read_joints() and write_joints(). Nonetheless, the C++ reference states that it's an operation with log complexity. The impact of these operations should be limited, because until know, we never used more than 18 actuators at the same time.
The alternative choice is to consider that the control loop cannot run at such frequency. I think it would be a sad limitation.
The text was updated successfully, but these errors were encountered: