Low frame rate and latency

[farhad-dalirani]

I have a Basler camera that is connected to my PC(24 GB RAM, Core i7) with a USB3. It is very slow and the frame rate is low. For example, I pass in front of it at t=6 (second), and it shows me on t=9 (second).
Also, the frame rate is so low although in the configuration file I determined the frame rate as 60.0.

I get this error on the terminal:

[ERROR] [1655490133.698416607]: Error: 3792703499 Not enough memory to submit transfer. You can increase the usbfs memory by modifying /sys/module/usbcore/parameters/usbfs_memory_mb.
[ERROR] [1655490133.698460259]: Error: Grab was not successful

Also, I get this warning on terminal:

[ WARN] [1655491176.729972777]: Did not reach the target brightness before timeout of 5 sec! Stuck at brightness -1.77634e+08 and exposure 100000us

This is my configuration file:

#  The tf frame under which the images were published
camera_frame: pylon_camera

#  The DeviceUserID of the camera. If empty, the first camera found in the
#  device list will be used
device_user_id: "ros_basler_left"

#  The CameraInfo URL (Uniform Resource Locator) where the optional intrinsic
#  camera calibration parameters are stored. This URL string will be parsed
#  from the ROS-CameraInfoManager:
#  http://docs.ros.org/api/camera_info_manager/html/classcamera__info__manager_
#  1_1CameraInfoManager.html#details
camera_info_url: ""

#  The encoding of the pixels -- channel meaning, ordering, size
#  taken from the list of strings in include/sensor_msgs/image_encodings.h
#  The supported encodings are 'mono8', 'bgr8', 'rgb8', 'bayer_bggr8',
#  'bayer_gbrg8' and 'bayer_rggb8'
#  Default values are 'mono8' and 'rgb8'
image_encoding: "rgb8"

#  Binning factor to get downsampled images. It refers here to any camera
#  setting which combines rectangular neighborhoods of pixels into larger
#  "super-pixels." It reduces the resolution of the output image to
#  (width / binning_x) x (height / binning_y).
#  The default values binning_x = binning_y = 0 are considered the same
#  as binning_x = binning_y = 1 (no subsampling).
binning_x: 1
binning_y: 1

#  The desired publisher frame rate if listening to the topics.
#  This parameter can only be set once at startup
#  Calling the GrabImages-Action can result in a higher framerate
frame_rate: 60.0

#  Mode of camera's shutter.
#  The supported modes are "rolling", "global" and "global_reset"
#  Default value is "" (empty) means default_shutter_mode
#shutter_mode: ""

##########################################################################
######################## Image Intensity Settings ########################
##########################################################################
# The following settings do *NOT* have to be set. Each camera has default
# values which provide an automatic image adjustment resulting in valid
# images
##########################################################################

#  The exposure time in microseconds to be set after opening the camera.
exposure: 100000.0

#  The target gain in percent of the maximal value the camera supports
#  For USB-Cameras, the gain is in dB, for GigE-Cameras it is given in so
#  called 'device specific units'.
# gain: 0

#  Gamma correction of pixel intensity.
#  Adjusts the brightness of the pixel values output by the camera's sensor
#  to account for a non-linearity in the human perception of brightness or
#  of the display system (such as CRT).
# gamma: 1.0

#  The average intensity value of the images. It depends the exposure time
#  as well as the gain setting. If 'exposure' is provided, the interface will
#  try to reach the desired brightness by only varying the gain. (What may
#  often fail, because the range of possible exposure vaules is many
#  times higher than the gain range). If 'gain' is provided, the interface will
#  try to reach the desired brightness by only varying the exposure time. If
#  gain AND exposure are given, it is not possible to reach the brightness,
#  because both are assumed to be fix.
brightness: 100

#  Only relevant, if 'brightness' is set:
#  The brightness_continuous flag controls the auto brightness function.
#  If it is set to false, the brightness will only be reached once.
#  Hence changing light conditions lead to changing brightness values.
#  If it is set to true, the given brightness will be reached continuously,
#  trying to adapt to changing light conditions. This is only possible for
#  values in the possible auto range of the pylon API which is e.g. [50 - 205]
#  for acA2500-14um and acA1920-40gm
brightness_continuous: true

#  Only relevant, if 'brightness' is set:
#  If the camera should try to reach and / or keep the brightness, hence
#  adapting to changing light conditions, at least one of the following flags
#  must be set.
#  If both are set, the interface will use the profile that tries to keep the
#  gain at minimum to reduce white noise.
#  The exposure_auto flag indicates, that the desired brightness will be
#  reached by adapting the exposure time.
#  The gain_auto flag indicates, that the desired brightness will be
#  reached by adapting the gain.
exposure_auto: true
gain_auto: true

##########################################################################

#  The timeout while searching the exposure which is connected to the
#  desired brightness. For slow system this has to be increased.
# exposure_search_timeout: 5.0

#  The exposure search can be limited with an upper bound. This is to prevent
#  very high exposure times and resulting timeouts.
#  A typical value for this upper bound is ~2000000us.
# auto_exposure_upper_limit: 2000000.0

#  The MTU size. Only used for GigE cameras.
#  To prevent lost frames configure the camera has to be configured
#  with the MTU size the network card supports. A value greater 3000
#  should be good (1500 for RaspberryPI)
# gige:
#  mtu_size: 3000

#  Only used for GigE cameras.
#  The inter-package delay in ticks to prevent lost frames.
#  For most of GigE-Cameras, a value of 1000 is reasonable.
#  For cameras used on a RaspberryPI this value should be set to 11772.
# gige:
#  inter_pkg_delay: 1000

[joako1991]

Hi,
I experienced similar issues when using a Kinect camera. The USB system contains a buffer in which the information is copied and then transferred to the devices. This buffer is the USBFS that the first warning mentions you. Depending on the number of devices you have connected to a single USB port and the architecture of your machine, you might need to increase your USBFS size. There are two possible ways to do it: a permanent solution or a temporal solution. Here you have the steps of how to do it in Linux: OpenKinect/libfreenect2#807 . A value of 64 in general is enough.

Regarding your frame rate, you have to pay attention that setting the FPS to a certain value does not necessarily imply that you will reach that FPS rate. It is a value related to the exposure time. If your exposure time is too high, then the frame rate cannot be high. For instance, you can get the maximum frame rate of 70 fps in some cameras only if your exposure time is less than 10 mS. This is because from your camera the image travels through the code that sometimes is not optimized, it might do some processing that will increase the time between frames, and any other operation you will do will also decrease the frame rate.

Another detail: When you retrieve an image from a Basler camera, you have to specify a timeout. If your exposure time is too high, and taking an image takes more time than the timeout time, then you will never receive an image from the camera (a timeout exception will always occur). Additionally, the configuration is persistent, so if you stop your program and you restart it, then the same situation will happen. What I used to do in my programs is to start the camera always with a safe configuration, that allows me to always receive images, and to always interact with the camera.

I hope this information is useful for you.

[m-binev]

@farhad-dalirani Hi, for the "usbfs" check the pylon ReadMe > Performance Optimization > USB Vision Cameras: https://www.baslerweb.com/fp-1654588449/media/downloads/software/pylon_software/README~5.txt

[farhad-dalirani]

The problem was solved.

The answer is in this StackOverflow post:

https://stackoverflow.com/questions/72671393/problem-with-basler-camera-low-frame-rate-and-latency-ros/72721932#72721932