/localization/util components die after init pose is set

[Travis-Bowman]

Checklist

• I've read the contribution guidelines.
• I've searched other issues and no duplicate issues were found.
• I'm convinced that this is not my fault but a bug.

Description

Once the initial pose is set, the vehicle appears correctly positioned and oriented on the map. The /sensing/lidar/concatenated/pointcloud topic (XYZIRC format) publishes data at approximately 10.5 Hz. Shortly afterward, the /localization/util/crop_box_filter_measurement_range, voxel_grid_downsample_filter, and random_downsample_filter components crash. As a result, the ndt_scan_matcher is unable to update the EKF.

Expected behavior

The expected behavior is that /localization/util/crop_box_filter_measurement_range, voxel_grid_downsample_filter, and random_downsample_filter components don't crash and ndt_scan_matcher updates EKF.

Actual behavior

The actual behavior is that /localization/util/crop_box_filter_measurement_range, voxel_grid_downsample_filter, and random_downsample_filter components crash and update counter on EKF keeps counting.

Steps to reproduce

1. ros2 launch autoware_launch autoware.launch.xml map_path:=ds_autoware_maps/research_drive vehicle_model:=pacifica_vehicle sensor_model:=pacifica_sensor_kit
2. Set initial pose in RVIZ
3. Watch EKF not update in rqt_robot_monitor

Versions

• OS: 22.04
• ROS2 Humble
• Autoware: main

Possible causes

Something with the pointcloud topic not being accepted. Initial pose is populating and that is the pose when the /localization/util pipeline dies.

Additional context

I am running a single Ouster lidar.

[YamatoAndo]

@Travis-Bowman
Thank you for the report.
I’d like to investigate the issue, so could you please share the following information?

• Dataset (rosbag, pcd map, etc.)
• Log file (~/.ros/log//launch.log)
• Screenshot of the RViz screen

[Travis-Bowman]

launch.log

I have uploaded the logs and screenshot, cannot seem to get a -a bag to upload.

[YamatoAndo]

@Travis-Bowman Thank you for the response.
I think the following error log is the cause.
As a result, the process of component_container_mt-1, which includes /localization/util/crop_box_filter_measurement_range, voxel_grid_downsample_filter, and random_downsample_filter, is crashing.

It seems to be an issue related to the GPU driver, but since I’m not very familiar with it, please check with the maintainer of this autoware_lidar_centerpoint or consult ChatGPT.

1756926009.9604537 [component_container_mt-1] terminate called after throwing an instance of 'std::runtime_error'
1756926009.9605904 [component_container_mt-1]   what():  cudaErrorUnsupportedPtxVersion (222)@/home/tbowman/Documents/dataspeed_autoware_future/autoware/src/universe/autoware_universe/perception/autoware_lidar_centerpoint/lib/centerpoint_trt.cpp#L222: the provided PTX was compiled with an unsupported toolchain.
1756926009.9606919 [component_container_mt-1] *** Aborted at 1756926009 (unix time) try "date -d @1756926009" if you are using GNU date ***
1756926009.9636514 [component_container_mt-1] PC: @                0x0 (unknown)
1756926009.9641407 [component_container_mt-1] *** SIGABRT (@0x3e8001f18c8) received by PID 2037960 (TID 0x79b40ba68f40) from PID 2037960; stack trace: ***
1756926009.9662974 [component_container_mt-1]     @     0x79b40b442520 (unknown)
1756926009.9685426 [component_container_mt-1]     @     0x79b40b4969fc pthread_kill
1756926009.9703751 [component_container_mt-1]     @     0x79b40b442476 raise
1756926009.9727399 [component_container_mt-1]     @     0x79b40b4287f3 abort
1756926009.9743965 [component_container_mt-1]     @     0x79b40b8a2b9e (unknown)
1756926009.9827347 [component_container_mt-1]     @     0x79b40b8ae20c (unknown)
1756926009.9838316 [component_container_mt-1]     @     0x79b40b8ae277 std::terminate()
1756926009.9838951 [component_container_mt-1]     @     0x79b40b8ae4d8 __cxa_throw
1756926009.9842992 [component_container_mt-1]     @     0x79b36062fa87 cuda::check_error()
1756926009.9856622 [component_container_mt-1]     @     0x79b36062e89f autoware::lidar_centerpoint::CenterPointTRT::preprocess()
1756926009.9906023 [component_container_mt-1]     @     0x79b36062dfeb autoware::lidar_centerpoint::CenterPointTRT::detect()
1756926009.9922836 [component_container_mt-1]     @     0x79b3609948b5 autoware::lidar_centerpoint::LidarCenterPointNode::pointCloudCallback()
1756926009.9924121 [component_container_mt-1]     @     0x79b3609dc03a std::__invoke_impl<>()
1756926009.9924529 [component_container_mt-1]     @     0x79b3609d84e1 std::__invoke<>()
1756926009.9924834 [component_container_mt-1]     @     0x79b3609d3b5a _ZNSt5_BindIFMN8autoware17lidar_centerpoint20LidarCenterPointNodeEFvRKSt10shared_ptrIKN15cuda_blackboard15CudaPointCloud2EEEPS2_St12_PlaceholderILi1EEEE6__callIvJOS7_EJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
1756926009.9925122 [component_container_mt-1]     @     0x79b3609ce676 std::_Bind<>::operator()<>()
1756926009.9925368 [component_container_mt-1]     @     0x79b3609c86cb std::__invoke_impl<>()
1756926009.9925709 [component_container_mt-1]     @     0x79b3609c0771 _ZSt10__invoke_rIvRSt5_BindIFMN8autoware17lidar_centerpoint20LidarCenterPointNodeEFvRKSt10shared_ptrIKN15cuda_blackboard15CudaPointCloud2EEEPS3_St12_PlaceholderILi1EEEEJS8_EENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESK_E4typeEOSL_DpOSM_
1756926009.9925942 [component_container_mt-1]     @     0x79b3609b7bdb std::_Function_handler<>::_M_invoke()
1756926009.9926164 [component_container_mt-1]     @     0x79b14ea5d13d std::function<>::operator()()
1756926009.9926369 [component_container_mt-1]     @     0x79b14ea58981 cuda_blackboard::CudaBlackboardSubscriber<>::compatibleCallback()
1756926009.9926589 [component_container_mt-1]     @     0x79b14eaa5189 std::__invoke_impl<>()
1756926009.9931183 [component_container_mt-1]     @     0x79b14eaa2a27 std::__invoke<>()
1756926009.9931550 [component_container_mt-1]     @     0x79b14eaa0104 _ZNSt5_BindIFMN15cuda_blackboard24CudaBlackboardSubscriberINS0_15CudaPointCloud2EEEFvRKSt10shared_ptrIKN11sensor_msgs3msg12PointCloud2_ISaIvEEEEEPS3_St12_PlaceholderILi1EEEE6__callIvJSD_EJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
1756926009.9931793 [component_container_mt-1]     @     0x79b14ea9cefa std::_Bind<>::operator()<>()
1756926009.9936454 [component_container_mt-1]     @     0x79b14ea981d1 std::__invoke_impl<>()
1756926009.9940221 [component_container_mt-1]     @     0x79b14ea901f7 _ZSt10__invoke_rIvRSt5_BindIFMN15cuda_blackboard24CudaBlackboardSubscriberINS1_15CudaPointCloud2EEEFvRKSt10shared_ptrIKN11sensor_msgs3msg12PointCloud2_ISaIvEEEEEPS4_St12_PlaceholderILi1EEEEJSE_EENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESO_E4typeEOSP_DpOSQ_
1756926009.9944112 [component_container_mt-1]     @     0x79b14ea82c89 std::_Function_handler<>::_M_invoke()
1756926009.9979796 [component_container_mt-1]     @     0x79b3d9945545 std::function<>::operator()()
1756926010.0013170 [component_container_mt-1]     @     0x79b3d99427c2 _ZZN6rclcpp23AnySubscriptionCallbackIN11sensor_msgs3msg12PointCloud2_ISaIvEEES4_E22dispatch_intra_processESt10shared_ptrIKS5_ERKNS_11MessageInfoEENKUlOT_E_clIRSt8functionIFvRKS9_EEEEDaSE_
1756926010.0032921 [component_container_mt-1]     @     0x79b3d9950d2e _ZSt13__invoke_implIvZN6rclcpp23AnySubscriptionCallbackIN11sensor_msgs3msg12PointCloud2_ISaIvEEES5_E22dispatch_intra_processESt10shared_ptrIKS6_ERKNS0_11MessageInfoEEUlOT_E_JRSt8functionIFvRKSA_EEEESE_St14__invoke_otherOT0_DpOT1_
1756926010.0067279 [component_container_mt-1]     @     0x79b3d994ae4e _ZSt8__invokeIZN6rclcpp23AnySubscriptionCallbackIN11sensor_msgs3msg12PointCloud2_ISaIvEEES5_E22dispatch_intra_processESt10shared_ptrIKS6_ERKNS0_11MessageInfoEEUlOT_E_JRSt8functionIFvRKSA_EEEENSt15__invoke_resultISE_JDpT0_EE4typeESF_DpOSO_
1756926010.7329917 [component_container_mt-80] [INFO] [1756926010.732579126] [planning.scenario_planning.lane_driving.behavior_planning.behavior_path_planner]: waiting for scenario_topic
1756926010.7568567 [component_container_mt-37] [WARN] [1756926010.755746768] [sensing.lidar.top.crop_box_filter_self]: 38987 points contained NaN values and have been ignored
1756926011.3018577 [ERROR] [component_container_mt-1]: process has died [pid 2037960, exit code -6, cmd '/opt/ros/humble/lib/rclcpp_components/component_container_mt --ros-args -r __node:=pointcloud_container -r __ns:=/ -p use_sim_time:=False -p wheel_radius:=0.365 -p wheel_width:=0.25 -p wheel_base:=3.08 -p wheel_tread:=1.54 -p front_overhang:=1.0 -p rear_overhang:=1.15 -p left_overhang:=0.26 -p right_overhang:=0.26 -p vehicle_height:=2.3 -p max_steer_angle:=0.6'].