Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Property Set commands are not propagated to camera via USB. #561

Open
c1arocque opened this issue Oct 16, 2024 · 9 comments
Open

Property Set commands are not propagated to camera via USB. #561

c1arocque opened this issue Oct 16, 2024 · 9 comments

Comments

@c1arocque
Copy link

Hello

I’m using a 72BUC02-ML in a personal 8mm film telecine project with the Python TIS() script. The application occasionally looses property set and software trigger commands. I have traces demonstrating that the function calls to tiscamera are executing and that no GStreamer errors are being raised. USB traffic capture (Wireshark) shows the missing commands are not being passed to the camera.

Camera configuration: BGRx, 2592 x 1944, frame rate == 5, trigger mode == set, all auto functions are disabled.

TISCamera built from github source this spring.

Linux laptop, 32 MB ram, M2 SSD, Ubuntu Studio 22.04, Python 3.

My application (script) captures a sequence of 5 exposures for each S8mm frame. The set of frame images are merged in a separate (from TIS()) process.

The film transport PCBA provides a hardware trigger for each frame.

Callback Function Outline.
If the callback is due to hardware trigger
# The first image of a new frame
Pull and Discard image
TriggerSoftware
exit
Else
Pull and Store image in exposure set array
Endif

If this is the last exposure of the set
Set exposure (first in sequence)
Pass set of frames to the merge process
else
Set exposure (next)
TriggerSoftware
endif
exit

The hardware trigger always generates a frame and executes the callback. I can see TIS()/GST() polling the camera interface prior to receiving a hardware triggered frame. How does TIS() know when to POLL? Does it use an estimate from a known camera (hardware) frame buffer size and the declared frame rate? I don’t see the same polling sequence when a software trigger is executed.

The setexposure and triggersoftware commands are occasionally not transmitted to the camera. When this occurs, the next hardware trigger event allows the completion of my exposure set. Of course the transport has advanced to the next frame and the exposure set is a composite of two frames.

It appears that the two missing commands are blocked and/or dropped. I have enabled logging for GST but there is no obvious indication of a root cause.

Please advise how I can resolve my issue.

Regards

Chris LaRocque
@TIS-Stefan
Copy link
Contributor

Hello Chris

I never had the situation you describe, which does not mean, it can happen. Is it possible to send a small script, which I can use to reproduce the situation on my computer?
However, if the situation is caused v4l2, then I am afraid, I can not help.

Stefan

@c1arocque
Copy link
Author

c1arocque commented Oct 17, 2024 via email

@c1arocque
Copy link
Author

c1arocque commented Oct 19, 2024 via email

@TIS-Stefan
Copy link
Contributor

Hello

I needed to reduce you program to the relevant parts, by removing all OpenCV / ImageIO etc stuff. Also I removed the multithreading part, because I am not sure, how v4l2 will handle this.
I took me some time to understand, what you are doing. The general idea of this HDR capture is quite good.
The reduced sample it this:

trigger-callback-softwaretrigger.zip

I simply looked on the live video and checked, whether the image brightness changes as wanted. This is always the case. Thus, I suppose, there is somewhat in the multithreading you use. Keep your code as simple as possible, then the you have less side effects.

Somewhat tricky was to figure out how to get the exposure times in the correct sequence, so the hardware trigger, which is the first trigger for a new frame creates an image with the first exposure time the exposure times array. Thus, there is some double code in the callback:

def on_new_image(tis, userdata: CustomData):
    """
    Callback function, which will be called by the TIS class
    :param tis: the camera TIS class, that calls this callback
    :param userdata: This is a class with user data, filled by this call.
    :return:
    """
    # Avoid being called, while the callback is busy
    if userdata.busy is True:
        return

    userdata.busy = True

    # Here we have the image with last set exposure time
    print(f"Current exposure {userdata.camera.get_property('ExposureTime')}")

    userdata.imagecount += 1  # Next image, next exposure time
    l = len(userdata.exposurelist)
    if userdata.imagecount < l:
        print(
            f"New exposure : {l}  {userdata.imagecount} {userdata.exposurelist[userdata.imagecount]}"
        )
        userdata.camera.set_property(
            "ExposureTime", userdata.exposurelist[userdata.imagecount]
        )
        tis.execute_command("TriggerSoftware")  # Send a software trigger
    else:
        userdata.imagecount = 0
        print(
            f"New exposure : {l}  {userdata.imagecount} {userdata.exposurelist[userdata.imagecount]}"
        )
        userdata.camera.set_property(
            "ExposureTime", userdata.exposurelist[userdata.imagecount]
        )

    userdata.busy = False

Please try the sample I posted above.

Stefan

@c1arocque
Copy link
Author

c1arocque commented Oct 21, 2024 via email

@c1arocque
Copy link
Author

c1arocque commented Oct 28, 2024 via email

@TIS-Stefan
Copy link
Contributor

Hello Chris

In this situation would guess the image data transfer is incomplete, so the frame is discarded. Please try a lower frame rate and check, whether the image data transfer is more reliable.
The loss of packages in the computer can be caused by CPU idle states or by high CPU load, which prevents the USB controller to save all data into memory.

@c1arocque
Copy link
Author

c1arocque commented Nov 2, 2024 via email

@TIS-Stefan
Copy link
Contributor

Hello Chris

I'm curious about the apparent contradiction of camera frame rate reduction
reducing missed software trigger commands. Can you expound/ clarify?

Reducing the frame rate reduces the bandwidth usage on the USB bus and creates longer times between image packets created by the camera. Thus, the PC's USB controller has more time to get the image data from the camera. The camera itself has only a little memory for image data in a ring buffer and if image data packets are not picked up in time, they are overwritten. This leads to a smaller than expected image buffers. tiscamera checks for the image buffer size and discards incomplete images. Therefore, sending a trigger to camera starts a new image and the image data is prepared to be picked up by the computer, but if the image is incomplete, it is dropped.

You can see the source code also in the "v4ldevice.cpp" ("v4l2device.cpp") standard sample, where you have

struct v4l2_buffer buf;

When reading frames you can check something like this:

int read_frame (void)
{
	CLEAR(buf);

	buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
	buf.memory = V4L2_MEMORY_MMAP;

	if (-1 == xioctl(fd, VIDIOC_DQBUF, &buf))
	{
		switch (errno)
		{
			case EAGAIN:
				return 0;

			case EIO:
				/* Could ignore EIO, see spec. */

				/* fall through */

			default:
				errno_exit("VIDIOC_DQBUF");
		}
	}

	assert(buf.index < n_buffers);
	if( _test == 0 )
	{
		_test = 1;
		//printf("Allocated buffer size : %d\n", buf.length );
	}

	// Check for complete images.
	if( buf.length !=  buf.bytesused)
	{
		//printf("Incomplete buffer: Expected: %d, received %d\n",  buf.length,  buf.bytesused );
		incompleteframecount++;
		snaperror = 2;
	}

	if (-1 == xioctl(fd, VIDIOC_QBUF, &buf))
		errno_exit("VIDIOC_QBUF");

	return 1;
}

That is a snippet from a sample I wrote a few years ago for checking frame drops. You will find similar code somewhere in the tiscamera repository.

In worst case, this leads to getting no images. In Windows, this situation can be caused by CPU Idle States and too high CPU load. In Linux, I saw this only on weak computer like Raspberry PI 3 and below. The simplest operations like copying a buffer from v4l2 buffer into another memory location leads to frame drops.

Newer camera models like DFK 32UR0521 have much more memory for image data packets, thus this issue is less probably. USB cameras like the 38U series have 128MB of RAM for images, so they can store the images until they are picked up completely.

I hope, this clarifies, what happens, as far as I understand this.

Stefan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@TIS-Stefan @c1arocque