Timer hanging and high CPU load when using MultiThreadedExecutor

[JasperTan97]

Bug report

Required Info:

• Operating System: Ubuntu 22.04

• Installation type: Binaries

• Version or commit hash: Iron

• DDS implementation: eProsima’s Fast DDS (the default)

• Client library (if applicable): rclpy

• CPU info (if needed):

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         39 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  12
  On-line CPU(s) list:   0-11
Vendor ID:               GenuineIntel
  Model name:            13th Gen Intel(R) Core(TM) i7-1355U
    CPU family:          6
    Model:               186
    Thread(s) per core:  2
    Core(s) per socket:  10
    Socket(s):           1
    Stepping:            3
    CPU max MHz:         5000.0000
    CPU min MHz:         400.0000
    BogoMIPS:            5222.40
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflus
                         h dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constan
                         t_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aper
                         fmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse
                         3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer
                          aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs
                          ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbas
                         e tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb 
                         intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves avx_vnni dtherm ida arat pln pt
                         s hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi umip pku ospke waitpkg
                          gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr
                          ibt flush_l1d arch_capabilities
Virtualization features: 
  Virtualization:        VT-x
Caches (sum of all):     
  L1d:                   352 KiB (10 instances)
  L1i:                   576 KiB (10 instances)
  L2:                    6.5 MiB (4 instances)
  L3:                    12 MiB (1 instance)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-11
Vulnerabilities:         
  Gather data sampling:  Not affected
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec rstack overflow:  Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW seque
                         nce
  Srbds:                 Not affected
  Tsx async abort:       Not affected

Steps to reproduce issue

My publisher:

import rclpy
from rclpy.node import Node
from std_msgs.msg import String
import time

class MyPublisher(Node):
    def __init__(self):
        super().__init__('my_publisher')
        self.publisher_ = self.create_publisher(String, 'my_topic',  10 ) 
        timer_period =  1 / 500  
        self.timer = self.create_timer(timer_period, self.timer_callback)

    def timer_callback(self):
        msg = String()
        msg.data = 'Hello, ROS2! ' 
        self.publisher_.publish(msg)

my subscriber:

import rclpy
from rclpy.node import Node
from std_msgs.msg import String

class MySubscriber(Node):
    def __init__(self):
        super().__init__('my_subscriber')
        self.subscription = self.create_subscription(
            String,
            'my_topic',
            self.listener_callback,
            10)
        self.subscription  # prevent unused variable warning

    def listener_callback(self, msg):
        pass

And my main function:

import rclpy
from my_multithreaded_package.publisher import MyPublisher
from my_multithreaded_package.subscriber import MySubscriber
import time

def main(args=None):
    rclpy.init(args=args)

    publisher = MyPublisher()
    subscriber = MySubscriber()

    executor = rclpy.executors.MultiThreadedExecutor(4)
    executor.add_node(publisher)
    executor.add_node(subscriber)
    executor.spin()

    # Shutdown the ROS client library
    rclpy.shutdown()

if __name__ == '__main__':
    main()

Expected behavior (which is what I get when using the SingleThreadedExecutor)

ros2 topic hz /my_topic
average rate: 499.945
        min: 0.002s max: 0.002s std dev: 0.00007s window: 501
average rate: 499.965
        min: 0.001s max: 0.002s std dev: 0.00008s window: 1001
average rate: 499.995
        min: 0.001s max: 0.003s std dev: 0.00009s window: 1502
average rate: 499.981
        min: 0.001s max: 0.003s std dev: 0.00009s window: 2002
average rate: 499.978
        min: 0.001s max: 0.003s std dev: 0.00009s window: 2502
average rate: 500.004
        min: 0.001s max: 0.003s std dev: 0.00009s window: 3003
average rate: 499.989
        min: 0.001s max: 0.003s std dev: 0.00010s window: 3503

Actual behavior

ros2 topic hz /my_topic
average rate: 25.933
        min: 0.002s max: 0.919s std dev: 0.16741s window: 29
average rate: 14.580
        min: 0.000s max: 0.937s std dev: 0.21933s window: 41
average rate: 13.928
        min: 0.000s max: 0.937s std dev: 0.21447s window: 62
average rate: 12.983
        min: 0.000s max: 0.937s std dev: 0.22058s window: 81
average rate: 10.575
        min: 0.000s max: 0.937s std dev: 0.24748s window: 85
average rate: 9.774
        min: 0.000s max: 0.937s std dev: 0.25675s window: 89
average rate: 9.845
        min: 0.000s max: 0.937s std dev: 0.25056s window: 102
average rate: 9.399

Additional information

So similar issues have been brought up with rclcpp, but I have not seen any comments made about rclpy. I found an issue here:ros2/rclcpp#1487, with other people also reporting something like: ros2/rclcpp#1618 and the fix is ros2/rclcpp#1516 and then ros2/rclcpp#1692.

Aside from the timer callback hanging (I assume it to be with after following some tic toc), my CPU load becomes really high using the MultiThreadedExecutor, while the SingleThreadedExecutor does not cause any noticeable CPU load. I have also tried using both the MutuallyExclusiveCallbackGroup and ReentrantCallbackGroup with no change in behaviour.

I am not sure if my QOS settings are the problem, or this is an issue intrinsic to python (because of GIL or etc.) but either a more suitable example for how to use the MultiThreadedExecutor could be provided (if my usage is wrong), or the ROS wiki pages should reflect that this significant problem exists (if no fix is possible).

Thank you for helping!

[domire8]

I have the same issue, +1 for this

[wjwwood]

From our weekly issue triage meeting:

Assigning @clalancette so he can make a high level tracking issue about known issues with Python's executors.

Also, we think this issue will not make progress without staffing of an engineer, which we don't currently have, or a dedicated community member to investigate and make a good suggestion on what to change and why. This is because executor related issues tend to be very nuanced and complicated to work on. So we're also assigning this the "help wanted" label with that in mind.

[KKSTB]

Hi @JasperTan97 I recommend you could run py-spy to see where the high CPU load comes from. This helps immensely. Thanks

[JasperTan97]

@KKSTB, it looks like the waiting for ready callbacks are the issue. Do you get the same on your end?

[KKSTB]

@JasperTan97 i ran py-spy on my ubuntu ROS iron machine and got this result:

And py-spy dump to see what threads they are in the flame graph:

Thread 11386 (idle): "MainThread"
    handle (rclpy/guard_condition.py:38)
    _wait_for_ready_callbacks (rclpy/executors.py:582)
    wait_for_ready_callbacks (rclpy/executors.py:723)
    _spin_once_impl (rclpy/executors.py:794)
    spin_once (rclpy/executors.py:813)
    spin (rclpy/executors.py:295)
    main (xavest_gui/main.py:44)
    <module> (xavest_gui:33)
Thread 11398 (idle): "ThreadPoolExecutor-0_0"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 11399 (idle): "ThreadPoolExecutor-0_1"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 11400 (idle): "ThreadPoolExecutor-0_2"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 11401 (idle): "ThreadPoolExecutor-0_3"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)

This shows the 4 worker threads were mostly idle, which makes sense because there is basically nothing to do inside subscriber and timer node.

The main thread was instead very very busy retrieving tasks for the 4 worker threads to do. It seems such workload of retrieving and distributing tasks and gathering results at high frequency (500Hz) is marginal for one core. Therefore the rate slows down considerably. Although I have no clue why your single core is much slower (I can achieve 3XX-4XX Hz on my i7-9750H).

As for single threaded executor, I can achieve 500Hz. The CPU utilization was half of multi threaded case. I have to push to 2500Hz before the rate starts to drop.

I believe the problem has to do with the efficiency of transferring tasks from main thread to worker threads, relative to the actual useful tasks done in the worker thread.

[MatthijsBurgh]

I experience the same issue, +1

[MatthijsBurgh]

SVG: https://gist.githubusercontent.com/MatthijsBurgh/a5ee1d710e0ff4b6524b279a2688e8be/raw/15da19a2ca8ca0d1e28bfeb46e9726205357a855/profile.svg

But it doesn't show the files and line numbers correct. Therefore I have included another screenshot, which does show this. You should check the linenumbers in this revision of executors.py

update: I have looked at the code a bit, _wait_for_ready_callbacks just looks very inefficient. It constructs many objects and lists. The lists are appended and then iterate over again, etc. The rclcpp::Executor, looks more efficient.
(It is complex code, so if you feel I am wrong, let me know.)

[clalancette]

update: I have looked at the code a bit, _wait_for_ready_callbacks just looks very inefficient. It constructs many objects and lists. The lists are appended and then iterate over again, etc. The rclcpp::Executor, looks more efficient.
(It is complex code, so if you feel I am wrong, let me know.)

Yep, you are correct. And this flame graph is the kind of thing I was waiting to see to confirm. So thanks for spending the time to do that.

We spent a bunch of time over the last year optimizing the rclcpp executor to make things faster; the rclpy one needs a similar effort to become more performant.

[MatthijsBurgh]

@clalancette just let me know whether you need a more detailed graph, etc.

[clalancette]

@clalancette just let me know whether you need a more detailed graph, etc.

To be totally transparent; we don't (currently) have plans to work on this. While that could change in the future, if it is something you are interested in, I'd suggest trying to make some improvements and opening PRs. We are more than happy to review them.