WebCam Capture Performance

[rakeshbhatt10]

Hi @sarxos ,

I had implemented your solution and tested on various processors i3 ( 4 GB RAM), i5 (4 GB RAM) , Core 2 Duo and Atom with different webcams. I had noticed that in every system FPS as well as Refresh rate gets changed. In some systems i had seen changes in application where FPS is good but refresh rate is slow. I had been trying with Logitech C920 webcam as external cam.

I didn't get why these changes happens. When i test Webcam with Logitech WebCam software It's performance of webcam preview remains constant in all systems even with high resolutions. I didn't get it.

It will be very helpful if you can provide your suggestions
Thanks in advance

[sarxos]

Hi @rakeshbhatt10 ,

Please clarify:

• What is the API version you was testing against?
• How was the tests executed (what was the light, etc)?
• What do you mean by Refresh rate?

[rakeshbhatt10]

Hi @sarxos

What is the API version you was testing against?
I am using your current API available for download that is webcam-capture-0.3.10-RC7 also i am using your default webcam driver

How was the tests executed (What was the light etc)?
There is normal indoor lighting conditions. I think Lighting is good.

What do you mean by Refresh rate?
Refresh Rate is repaint per second

Additional info:

1. Resolution in which i tested 640x480, 1024x768, 960x720 I had also tried with 1600x1200 and 1920x1080 worse results on low processors like Atom and Core 2 Duo.,
2. Code which i am using for attaching webcam panel:

Dimension requiredDimension = new Dimension(resX,resY);
Dimension[] dime = {requiredDimension};
Webcam webCam = Webcam.getWebcams().get(cameraid);
webCam.setCustomViewSizes(dime);
webCam.setViewSize(requiredDimension);
WebcamPanel webCamPanel = new WebcamPanel(webCam);
webCamPanel.setFPSDisplayed(true);
webCamPanel.setDisplayDebugInfo(true);
webCamPanel.setImageSizeDisplayed(true);
webCamPanel.setMirrored(true);
contentPane.add(webCamPanel,BorderLayout.CENTER);

I had tried with webCamPanel.setMirrored(false) or removed it but there is no difference

1. I am using Jdk 6.

I want to know the reason why this actually happens. I had been hanging with these issues throughout the development

Thanks,

[sarxos]

HI @rakeshbhatt10,

Thank you for the clarification.

The FPS should be pretty stable between the operating systems whenever application is run on the machine with comparable processing power. It depends on the following aspects:

• Source of light - the more light, the higher FPS is,
• Processing power - better processor and more cores can handle I/O operations faster and causes higher FPS,
• GPIO speed - in case of slow GPIO, the FPS will be lower as well. This can be observed e.g. on RasPi,
• Image resolution - the higher resolution is, the more bytes need to be processed through GPIO, more bytes needs more time, which lowers FPS.
• The camera itself - various devices may behave differently - different hardware may stream max 25, 30, 35 or even 50 FPS (most common is 30).

The refresh rate is a average speed of how WebcamPanel is being rendered. This value depends on the following:

• Computer processing power - the higher power is, the faster bytes are transferred from RAM to video memory, this causes higher refreshing rate.
• The graphic card processing power and various types of acceleration it supports - Java strongly depends on the acceleration support which specific graphic card can offer it. The acceleration is done under the hood and the only think which we can do, is to avoid touching it, so it's not broken by accident (see Slowdown on resize between RC6 and build of June 15, 2014 #223 and [Request] Fix horizontal direction. #227 which fixed one of the latest rendering issues I accidentally made).
• Operating system - various acceleration and graphic rendering method are used (via DirectDraw pipeline) on different platforms - Java on Linux uses calls to OpenGL, on Windows it uses Direct3D (?), but depending on the graphic drivers and hardware, the different acceleration mechanisms may be supported or not, which causes higher or lower rendering performance. Generally, basing on what I see on my dev environment, Java on Linux (at least Ubuntu 12) has much better acceleration support than the same on Windows. I have no idea what is used on Mac since I'm completely unfamiliar with this OS.

If you are curious of how Java rendeing/acceleration works:

http://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-Desktop/html/java2d.html

I hope the above clarifies your doubts.

[rakeshbhatt10]

Hi @sarxos

Thanks a lot for providing your inputs on this problem

I am completely agree @sarxos with your points on Processing Power, GPIO speed and Image Resolution. You had described them very well.

"I want to ask that how under same lighting condition on low processors Logitech WebCam software provides much more better webcam preview output."

There is no lagging nothing at all with higher resolution too with Logitech WebCam Software. i have seen the same capability in Skype as well with Logitech webcams.

As you have describe the points which are reason for slow webcam preview. Is there can be any better way to achieve the same performance as Logitech WebCam Software in Java ?

[sarxos]

@rakeshbhatt10, in regards to Logitech WebCam Software, I can only have my own suspicions, can only guess why this software is faster. Those are my more or less probable thoughts:

• It operates directly on RAW image data in RAM. Java needs one more step to process RAW data to format which can be consumed by Java API (it needs single copy data in RAM). This takes some time, small amount, but in case of low processing power, it can be greater.
• It works on different abstraction layer, using different libraries. In case of Webcam Capture API, every call to webcam hardware is delegated to BridJ, which invokes JNI, which invokes OpenIMAJ natives, which invokes VideoInput library, which invokes DirectX API, which finally build capture graph, which calls camera drivers. Logitech WebCam Software invokes different API. I don't know what is used beneath - maybe DirectX, maybe Media Foundation, however, when dealing with native applications, you can choose faster programming apparatus in exchange for portability and/or complexity.
• There is a possibility that Logitech WebCam Software access some camera controls which Webcam Capture is unaware of, and those controls may somehow increase FPS (e.g. some parameter to control image sensor sensitiveness). Webcam Capture API does not have any possibility to access these additional controls because of the different operation abstraction layer.

[rakeshbhatt10]

Thanks @sarxos for this explanation

I am also trying to use another camera from different vendors. I had tried with other cameras from e-consystesm in this link
http://www.e-consystems.com/5mp-usb-cameraboard.asp I can see there is separate fps for both YUV and MJPEG. Tried looking into logs and methods but couldn't find in which format Webcam-capture API shows webcam live stream.

As per the hardware specification FPS rate is higher in MJPEG format
-------------------- YUV2---------------- MJPEG
VGA---------------30 fps----------------30 fps
HD (720p)--------7.5 fps--------------- 30 fps
QXGA (3M)---------2 fps--------------- 15 fps
QSXGA (5M)-------2 fps------------- --15 fps

I had tried below code to capture MJPEG from webcam checked in chrome. Performance is worse

Webcam webCam = Webcam.getWebcams().get(1);
Dimension[] prefDimension ={ new Dimension(640,480)};
webCam.setCustomViewSizes(prefDimension);
webCam.setViewSize(prefDimension[0]);
WebcamStreamer web=new WebcamStreamer(8080, w, 0.5, true);
do {
    Thread.sleep(5000);
} while (true); 

There is any way i can integrate MPEG with WebCamPanel. Will it increase any performance of webcam capture on low processors ?

[sarxos]

Hi @rakeshbhatt10,

Unfortunately for you, Webcam Capture API in all cases uses YUV format. The example you posted reads the RAW image from camera in YUV format and transcode it to MJPEG stream used by web browser.

The MJPEG support has been planned as enhancement #145, but I never had enough time to implement it (constantly busy at work) :(

[cosmac]

Hi @sarxos,

Is there a way to access directly to the RAW image from camera ? I need access to original format YUV422i, that my camera streams. (I want to avoid RGB conversion)

Thank you for your time.

Andrei

[sarxos]

Hi @cosmac,

With existing implementation I'm afraid this is not possible.

[cosmac]

Hi @sarxos ,

Thank you for your fast reply. It is almost impossible to find something like that :)

OpenCV states that you should be able to save YUV format, but actually it is not implemented. All the frames are converted to RGB no matter what format comes from cameras.

Thank you again!

[sarxos]

Hi @cosmac,

From what I remember OpenCV uses slightly modified VideoInput lib which is also used by the Webcam Capture API default driver. Unfortunately, it chooses format automatically (loop through the list and select RGB24).

However, you could try Java bindings for GStreamer. There is a version compatible with GS 0.10.x available from Google Code (please note people reported having problems finding 64-bit version of this stuff, but if you decide to use 32-bit you have to have 32-bit Java as well):

https://code.google.com/p/gstreamer-java

It's used by my GStreamer capture driver (see webcam-capture-driver-gstreamer) so in any case you do not have to start from scratch.

There is also version compatible with GS 1.x available here on Github. It uses one of the most recent GStreamer releases:

https://github.com/gstreamer-java/gst1-java-core

There is also example demonstrating how it can be used with webcam:

https://github.com/gstreamer-java/gst1-java-examples/tree/master/src/main/java/org/freedesktop/gstreamer/examples

In regards to GS 1.x I'm planning to prepare dedicated capture driver to replace previous old 0.10.x but first I need to handle stuff at work so I have some spare weekends to donate into the project (games industry is tough sometimes).

Just for your information, in GStreamer Java code you can iterate over Caps and check the format (this is code for 0.10.x but I guess that 1.x is very similar or even simplified):

// create and bind everything together (pipeline, source, sink, filters, etc)
Pad pad = source.getPads().get(0)
Caps caps = pad.getCaps();
for (int i = 0, n = caps.size(); i < n; i++) {
  System.out.println(caps.getStructure(i).getName()); // e.g. "video/x-raw-yuv"
}

Also you can create Caps from String:

String format = ... // find the format matching YUV422i
Caps caps = Caps.fromString(String.format("%s,width=%d,height=%d", 
format, size.width, size.height));
filter.setCaps(caps); // this will cause given format to be used

Here is how I'm doing this, but this still takes RGB as a priority (YUV gave me less FPS so I'm using it in second place if there is no RGB support):

https://github.com/sarxos/webcam-capture/blob/master/webcam-capture-drivers/driver-gstreamer/src/main/java/com/github/sarxos/webcam/ds/gstreamer/GStreamerDevice.java

I would lie if I say that GStreamer impl is easy but if you choose this path I'm sure GStreamer Java community will help you.

[cosmac]

Hi @sarxos ,

Thank you so much for your detailed feedback. I will follow this path and see if I can achieve something.

[peterchaula]

@sarxos if you feel you have too much on your plate please feel free to assign some of the little issues to me. :)

[sarxos]

@peterchaula please feel free to select any issue you want and send pull request for it. I will be happy to review :)

[simon1389]

Hey!
I'm using the WebcamCapture API for my Java Application, because it's easy and straight forward.

Now i'm having a performance issue which i cannot solve, and i'm not sure if it is possible at the moment.
I'm developing on macOS. I got a webcam which supports 1920x1080 resolution and about 30 FPS at this resolution with MJPG (only 6 FPS with yuv). If i start the VLC Player directly and start a stream with the webcam as source, the performance is really good and the stream is not laggy.
If i start the WebcamPanel Example, and set the resolution to 1920x1080, it becomes really laggy. About 5 FPS.
I already tried to use different drivers, and the only driver working was the VlcjDriver with the workaround to directly pass the MediaListItems to the driver. The performance was not better, because i think should set some settings in the driver which are then passed to vlc, but i dont know how to do this. I think i should set some different codec like MJPG or similiar?
Now i'm trying the GStreamerDriver, but i dont get the gstreamer-lib 0.10 running.

So... the WebcamCapture API is really nice, and perfectly fits my needs. The only problem is this performance issue i spent already 2 days on.
Could anyone help me with that?

Edit: i don't get 5 FPS, but the repaints per second are about 5. So, the FPS (shown in the webcampanel) stay constant about 30. But the repaints are getting lower, the higher i set the resolution