-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Video Preview and Video Recording Classes
The following are some of the JavaCV classes used to show and record a video from a video camera linked to the computer. There are some basic generic classes used as general line, those are extended to create different classes based on the specific needs. The following are based on the study of the source code and some experiment with Java.
Frame is the frame of a video, and it encompasses both the image data and the data of the audio. It is used by the classes CanvasFrame (shows the user), FrameGrabber (get the frame from a video or a webcam), FrameRecorder (stores it on the hard disk), and their subclasses. By using FrameConverter you can convert a frame into other types of images as Bitmap (Android), Mat (OpenCV) or BufferedImage (Java) and vice versa.
The class is basically a set of data associated with the frame:
- public int imagewidth, imageHeight, imageDepth, imageChannels, imageStride; width, height, number of bits associated with each channel (there are various constants DEPTH_ *), number of channels and stride, or the number of bytes that separate the beginning of a line of the image and the beginning of the next line, It is typically equal to the width plus the number of bytes used as padding, which influence the way in which the image is saved in memory
- public buffer [] image; Buffer that contains the pixels of one of the channels of the image, but in the constructor is an array is created with length 1, despite what I think it has a buffer for each channel of the image. Based on the value of the Buffer imageDepth created is prepared for a data type specifically (ByteBuffer, ShortBuffer, IntBuffer, DoubleBuffer, ...)
- public int SampleRate, audioChannels; are information related to the variable buffer [] samples, are probably the sampling frequency of and the number of audio channels of audio associated with frame
- public buffer [] samples; Buffer containing audio samples, it is an array, and each element is an audio channel
- public Object opaque; an Object to Java, the object contains the underlying data "The underlying data objects, for example, AVFrame, IplImage, or Matt." I think it is used when the frame is created from an image of another type ( like Mat OpenCV)
- public long timestamp; the time stamp associated with the frame at the time of its creation, the time at which it was created (in milliseconds since January 1, 1970)
Implements the interface Indexable of JavaCPP, you need to access the data object through a Indexer and its subclasses, providing access multidimensional (stuff related to large amounts of data) easy and efficient array of primitive data (int, byte, float, ...) and NIO Buffer (Not Blocking Input / Outbut they are Java's Buffer). In addition to this class contains only three methods, one belongs to the interface Indexable, the others are clone() for the entire class or just for the buffer.
It is an abstract class with many subclasses: DC1394 Frame Grabber, FFmpegFrameGrabber, FlyCapture2FrameGrabber, FlyCaptureFrameGrabber, IPCameraFrameGrabber, OpenCVFrameGrabber, OpenKinectFrameGrabber, PS3EyeFrameGrabber, VideoInputFrameGrabber. They are all part of the library javacv. Basically consists of a set of data, all relating to the video, most of the methods are the getters and setters. It contains abstract methods for its subclasses (start, stop, grab, triggers, grabFrame). It also contains a list with all the names of the subclasses that extend it. There is no documentation for this class, neither for its subclasses.
This class subclasses, can be used to obtain a preview from one or more camera, to reproduce a video, and togheter with the class FrameRecorder to create a new video from the camera or from an existing file.
By performing the method grab(), the FrameGrabber loads the next frame of video, and saves within its global variable data for that frame.
The first methods are static getDefault(), get (String className), create (Class <? Extends FrameGrabber> c, p Class, Object o), linked to a variable List list that contains the names of all subclasses of FrameGrabber ("VideoInput", "OpenCV", "FFmpeg", "IPCamera", ...). These methods are based on the Java's class Method, which allows you to use the methods of a class even if in the code is not written which class it is. Probably they are used to create a subclass of FrameGrabber programmatically. To the create method, there are various methods create() associated, that allow you to create a FrameGrabber indicating the name of the class and where to take the video (file name, camera number, etc ..) and methods createDefault(), in which the name of the class is achieved with getDeafult().
There is also an init () method, which calls the tryLoad for all subclasses. However it is never called in the class.
The variables of the class are the following, for each there is a Getter and a Setter:
- VideoStream protected int = -1, audiostream = -1;
- protected String format = null;
- imagewidth protected int = 0, = 0 imageHeight, audioChannels = 0;
- protected ImageMode imageMode = ImageMode.COLOR; ImageMode is an enumerator of FrameGrabber, it may be worth COLOR, GRAY or RAW
- protected long sensorPattern = -1L;
- PixelFormat protected int = -1, videocodec, videoBitrate = 0;
- protected double aspectRatio = 0, frameRate = 0;
- sampleFormat protected int = 0, Audiocodec, audioBitrate = 0, SampleRate = 0;
- triggerMode protected boolean = false;
- protected int bpp = 0;
- protected int timeout = 10000;
- numBuffers protected int = 4;
- protected double gamma = 0.0;
- deinterlace protected boolean = false;
- protected HashMap <String, String> options = new HashMap <String, String> ();
- protected HashMap <String, String> videoOptions = new HashMap <String, String> ();
- protected HashMap <String, String> audioOptions = new HashMap <String, String> ();
- protected HashMap <String, String> metadata = new HashMap <String, String> ();
- protected HashMap <String, String> videoMetadata = new HashMap <String, String> ();
- protected HashMap <String, String> audioMetadata = new HashMap <String, String> ();
- frameNumber protected int = 0; the number of the current frame, in that position is
- protected long timestamp = 0; indicates the time stamp of the current frame, that is, how long after the start of the video that image was acquired in millionths of a second
There are abstract methods: start(), stop(), trigger(), grab(), grabFrame() and release(), plus a couple of very simple methods that restart() (run stop and then start) and flush() (performs a grab inside a for loop). All these methods can throw a FrameGrabber.Exception.
After those there are an object ExecutorService and a Future, related to a series of methods "delayed", the main is delayedGrab(final long DelayTime), that seems to capture a frame after a certain delay. Actually, it captures frames continuously, until it exceeds a delay equals to the parameter delayTime, then does nothing and the last frame captured remains stored in the variable Frame delayedFrame. To get the delayedFrame, ie the frame captured after waiting for a time equal to DelayTime, use the method getDelayedFrame(): This returns the result of the last delayedGrab made (null if it has never been done), using the method future.get (), which waits until the delayedGrab not been completed, so it is a method that can block the program.
Finally, Frame Grabber also contains inner classes:
- Frame Grabber.Array: it is capable of handling an array of FrameGrabber, it contains all the methods start, stop, etc ... of FrameGrabber, which call the methods of the same name for each of the FrameGrabber array. The method grab() instead do additional work, because estimated delays between frames of various FrameGrabber finally returns an array with all the captured frames and latency information. If the array consists of FrameGrabber of different types (eg. OpenCVFrameGr and FFMpefFrameGr) assumes that can not be synchronized between them, so there is no calculation on latency
- FrameGrabber.Exception: the exception thrown when there is a problem in FrameGrabber, it does not have anything other than normal Exception of Jav
- FrameGrabber.ImageMode: an enumerator used for variable imageMode, it may be COLOR, GRAY or RAW
- FrameGrabber.PropertyEditor: a class that extends PropertyEditorSupport. The Java Bean are Java classes that are only a set of variables, and contain only methods Getter and Setter written in a format getNameVar setNameVar, they also implement Serializable. The PropertyEditor allows you to make JavaBeans simpler, allowing you to get the names of the variables as a list of String and use those to get and set the value.
DC1394FrameGrabber is a class that extends the FrameGrabber of JavaCV, it can capture frames from a video camera or a video file. It uses the methods of the libraries javacpp.opencv_core, javacpp.opencv_imgproc and javacpp.dc1394. It is based on the library libdc1394, used to communicate with all the cameras that support the IEEE 1394 standard.
FFmpegFrameGrabber is a class that extends the FrameGrabber of JavaCV, it can capture frames from a video file. It uses the methods of the libraries javacpp.avcodec, javacpp.avdevice, javacpp.avformat, javacpp.avutil and javacpp.swscale, it is based on the FFmpeg library.
The methods createDefault() create a new object with new FFmpegFrameGrabber() and return it.
The method getDeviceDescriptions () method calls the tryLoad () using the Loader.load () of javaCPP, but it seems that will not be called automatically. The method tryLoad () instead is called automatically thanks to static {tryLoad ();}, call Loader.load for all classes before seen (avutil.class, avcodec.class, etc ... plus also javacpp.swresample.class ). Then records all formats and codecs with the methods avcodec_register_all(); av_register_all (); avformat_network_init (); avdevice_register_all ();.
Constructors are used to set the source from which to obtain the frame, there are 2 of them, and they receive: String filename path location of the file; File file that calls the String constructor, passing the value file.getAbsolutePath (). Constructors do not do anything, they just store the variable. It is not possible to record from a video camera.
The method grabFrame(boolean doAudio, doVideo boolean, boolean processing, boolean keyframes) is called by various types of grab. After each grab, the grabber class keeps some of the data of the last frame obtained, like the timestamp and the number that identifies the position of the frame in the video:
- Public Frame grab() throws Exception {return grabFrame (true, true, true, false); } Returns frames without any processing. What happens in practice is that most of the frames are returned empty, they're not null, but with all the parameters setted to 0, then after a certain period are returned a number of correct frames (probably the keyframes, which are saved as images) This alternation of empty frames and keyframes is repeated throughout the video. Probablythose are the data as stored in the file, it does not reconstruct the intermediate frames, and display them in the order they are stored.
- Frame grabImage() throws Exception {return grabFrame (false, true, true, false); } This is the method that returns in order all frames of the video
- Frame grabSamples() throws Exception {return grabFrame (true, false, true, false); } Returns in order all frames of the video, but they are all empty (their parameters are 0)
- Frame grabKeyFrame() throws Exception {return grabFrame (false, true, true, true); } Returns in order only the keyframes, skipping the others
FlyCaptureFrameGrabber is a class that extends the FrameGrabber of JavaCV, it can capture frames from a video camera or a video file. It uses the methods of the libraries javacpp.opencv_core, javacpp.opencv_imgproc and javacpp.PGRFlyCapture. It is based on the SDK PGR FlyCapture, used by all PointGrey's cameras.
The methods createDefault() create a new object with new FlyCaptureFrameGrabber() and return it. However, if you try to create a FlyCaptureFrameGrabber from a file, it will generate an exception.
The method getDeviceDescriptions() method calls the tryLoad () using the Loader.load () of javaCPP, but it seems that will not be called automatically. He does other things using the method flycaptureBusCameraCount(count) and the class javacpp.PGRFlyCapture
The constructors only accept the identifier of the camera, they call the methods flycaptureCreateContext(context) and flycaptureInitializePlus(context, DeviceNumber, numBuffers, (BytePointer) null).
OpenCVFrameGrabber is a class that extends the FrameGrabber of JavaCV, it can capture frames from a video camera or a video file. It uses the methods of the libraries javacpp.opencv_core, javacpp.opencv_imgproc and javacpp.opencv_videoio, it is based on the OpenCV library.
The methods createDefault() creates a new object with new OpenCVFrameGrabber() and return it.
The method getDeviceDescriptions () method calls the tryLoad () using the Loader.load () of javaCPP, but it seems that will not be called automatically.
Constructors are used to set the source from which to obtain the frame, there are 3 of them, receiving different parameters: int device, it receives an integer that identifies a video camera to be recorded; String filename, which receives the location to find the file; File file, which invokes the constructor String passing file.getAbsolutePath(). Constructors don't do anything, they just store the variable.
VideoInputFrameGrabber is a class that extends the FrameGrabber of JavaCV, it can capture frames from a video camera. It use the methods of the libraries javacpp.opencv_core, javacpp.opencv_imgproc and javacpp.videoInputLib.
The methods createDefault() create a new object with new VideoInputFrameGrabber() and return it, but if you try to create a VideoInputFrameGrabber from a file will generate an exception.
The method getDeviceDescriptions () method calls the tryLoad () using the Loader.load() of javaCPP, but it seems that will not be called automatically.
Constructors are used to set the source from which to obtain the frame, there is only one: int device, receives an integer that identifies a video camera to record from. Constructors do not do anything, they just store the variable.
It is an abstract class with few subclasses: FFmpegFrameRecorder and OpenCVFrameRecorder. They are all part of the javacv library. It basically consists of a set of data, all relating to the video, most of the methods are the getters and setters. It contains abstract methods for its subclasses (start, stop, record, release). It also contains a list with all the names of the subclasses that extend it. There is no documentation for this class, neither for its subclasses.
This class can be used to create a ne video file from a set of images. Togheter with a FrameGrabber it can be used to record a video from a camera, or to merge differents video files.
The first methods are static getDefault(), get(String className), create(Class <? Extends FrameRecorder> c, p Class, Object o), linked to a variable List list that contains the names of all subclasses of FrameRecorder ("FFmpeg" and "OpenCV"). These methods are based on the Method class of Java, which allows you to use the methods of a class even if in the code is not written which class it is. Probably they are used to create a subclass of FrameRecorder programmatically. To the create method are associated various methods create(), that allow you to create a FrameRecorder given the name of the class and where to take the video (file name, etc ..) and methods createDefault (), in which the name of the class is achieved with getDeafult ().
There is also an init() method, which calls the tryLoad for all subclasses. However it is never called in the class.
The variables of the class are the following, for each there is a Getter and a Setter:
- protected String format, videoCodecName, audioCodecName; name and format of audio and video codecs
- protected int imagewidth, imageHeight, audioChannels; width, height and number of audio channels in the video
- protected int PixelFormat, videocodec, videoBitrate, gopSize = -1; pixel format, that is, what are the data (RGB or BGR or HSV) and how many bits is any data format, video codec, bit rate at which the video is saved, and Group Of Pictures, the order in which the intra frames and Inter are arranged, emit one intra frame every gopSize frames at most
- protected double aspectRatio, frameRate, videoQuality = -1;
- protected int sampleFormat, Audiocodec, audioBitrate, SampleRate;
- protected double audioquality = -1;
- protected boolean interleaved; Interleaving in Computer Science is a way to arrange data in a noncontiguous way to increase to performance, Lets look at it like this. You have one video stream and one (matching) audio streams and you want Those streams put together in one file. One way of doing it would be to paste the audio stream behind the video stream. However there is a problem with this: if the streams get big your read device constantly has to search back and forward in the file; you want video and audio played in synch know it has to read a part of the video and then a part of audio That matches also for CDs / DVDs constanlty seeking back and forward in the file is not really recommendable. The obvious solution to this problem would be to break up the stream in chunks (for a video stream in chunks is one video frame) and put chunks of video and audio That Should Be played blackberries or less together, close to each other in the file . This way it is much easier to read out the file. For example: a video stream of frames 6:
V0 V1 V2 V3 V4 V5and a matching audio stream of 4 blocks:A0 A1 A2 A3:
A non-interleaved file:V0 V1 V2 V3 V4 V5 A0 A1 A2 A3;
Interleaved:A0 V0 V1 V2 V3 A1 A2 A3 V4 V5 - protected HashMap <String, String> options = new HashMap <String, String> ();
- protected HashMap <String, String> videoOptions = new HashMap <String, String> ();
- protected HashMap <String, String> audioOptions = new HashMap <String, String> ();
- protected HashMap <String, String> metadata = new HashMap <String, String> ();
- protected HashMap <String, String> videoMetadata = new HashMap <String, String> ();
- protected HashMap <String, String> audioMetadata = new HashMap <String, String> ();
- frameNumber protected int = 0;
- protected long timestamp = 0;
There are abstract methods: start(), stop(), record() and release(). All these methods can throw a FrameRecorder.Exception.
FFmpegFrameRecorder is a class that extends the FrameRecorder of JavaCV able to save a video file by adding frames. It uses the methods of the libraries javacpp.avcodec, javacpp.swresample, javacpp.avformat, javacpp.avutil and javacpp.swscale.
The methods createDefault() create a new object with new FFmpegFrameRecorder() and return it.
There are several methods that begin with records:
- record (Frame frame, int PixelFormat), it's the method that is called from the normal record(). If the frame is null calls the recordImage () passing 0 for all values of the image. Otherwise if the Buffer frame.image exist, calls the recordImage () passing the image data, and if the buffer frame.samples exist, calls the recordSamples passing the audio data
- boolean recordImage (int width, int height, int depth, int channels, int stride, int PixelFormat, Buffer ... image), is the method that saves the video image.
- boolean recordSamples (Buffer ... samples), calls the recordSamples (0, 0, samples)
- boolean recordSamples (SampleRate int, int audioChannels, Buffer ... samples), is the method that deals with saving the video sound. Finally it calls the method records (AVFrame)
- boolean records (AVFrame frame), AVFrame FFMpeg is a structure that contains audio and video data in raw form.
OpenCVFrameRecorder is a class that extends the FrameRecorder of JavaCV able to save a video in memory, adding to it a frame at a time. It uses the methods and libraries javacpp.opencv_core and javacpp.opencv_videoio.
The methods createDefault() creates a new object with new OpenCVFrameRecorder() and return it.
The method tryLoad () use the Loader.load() of javaCPP, but it seems that it's not called automatically. Is called in the static{} of FrameRecorder, then probably it is called automatically at the beginning of the program.
Constructors must set the File where to save the video (String filePath or File files, thar uses the String constructor with file.getAbsolutePath()) and the width and height of the video. The frame rate is set to 30, the size of the pixels is set to 1, and the video codec is set as opencv_videoio.FOURCC_* using constants (the value depends on a Windows variable, a global constant that indicates whether we are on a Windows operating system, it's CV_FOURCC_PROMPT for windows and CV_FOURCC_DEFAULT otherwise (it should be only for linux)). FourCC is a code used to indicate the AVI video codec. Constructors don't do anything, they just store the variable.
It also contains the abstract methods of FrameRecorder:
- start(): creates the writer object with the method cvCreateVideoWriter(filename, videocodec, frameRate, cvSize(imagewidth, imageHeight), PixelFormat), throws an exception if the object was not created. The method cvCreateVideoWriter creates the object VideoWriter of OpenCV, and it probably also calls the open() that initializes writing
- stop(): calls the release()
- record(Frame frame): transforms the frame in a IplImage, which is saved using the method cvWriteFrame(writer, image). At the end of the method, the frame is marked as keyframe (frame.keyFrame = true, always occurs unless an exception occurs)
- release(): Releases the writer with cvReleaseVideoWriter(writer), then it become null It also contains the finalize() method of the Object class of java, which calls the Object.finalize() and then release().
It doesn't contains getter and setter methods, those used belong to the superclass FrameRecorder.
A FrameConverter allows to easily convert an image type used in a certain API, in another type of image used in another API. For example you can transform OpenCV's Mat in an Android's Bitmap. JavaCV uses the Frame class, which is an image without relying on Android, OpenCV or other, you can obtain it with a FrameGrabber and show it with a CanvasFrame, by using OpenCVFrameConverter you can convert the frame into a mat, modify it with OpenCV and then convert it back to a frame.
The class contains only two methods both called convert(), which convert the frame in a certain type of image. Subclasses are AndroidFrameConverter (for Bitmap), Java2DFrameConverter (for BufferedImage, it contains many methods) and OpenCVFrameConverter (for Mat and IplImage).
CanvasFrame is a class that extends Java's JFrame that allows you to have a video which show the Frame. It has a subclass GLCanvasFrame.
ATTENTION: Make sure OpenGL or XRender is enabled to get low latency, something like
export _JAVA_OPTIONS -Dsun.java2d.opengl = = True
export _JAVA_OPTIONS -Dsun.java2d.xrender = = True
It has various constructors, all receive the title for the JFrame, you can also assign a value range (used to change the contrast of the image, the monitor by their nature change the contrast of the image, this value range is used to compensate this change, to be able to see the image with the correct contrast). It can receive an object GraphicsConfiguration (an object that contains all the information of a device such as a monitor or printer), a Java object DisplayMode (contains width, height, bit depth and frequency of redrawing an object GraphicsDevice) and number of screens to display.
There is a private method that is called by the constructor, init(final boolean fullScreen, DisplayMode displayMode final, final double range). This method in turn calls the method initCanvas(boolean fullScreen, DisplayMode displayMode, double range).
The init() method gets the information on the screen, and prepare to show at fullscreen and with correct range. Finally initCanvas calls the method, which creates an object Canvas and adds it to the JFrame, there are also the update() method (which calls the paint() method) and paint() method, which draws the image, if there is a variable color the CanvasFrame draws the color, then if there is a variable image it draws the image, finally if there is a variable buffer it draws the image.
The basic method is showImage(), which receives the image to display on the screen:
- public void showImage(Frame image) {showImage(image, false); }
- public void showImage(Frame image, boolean flipChannels): transforms the image frame into a BufferedImage using Java2DConverter, taking into account the value of inverseGamma and flibChannels
- public void showImage(Image image): this is the method in which there is the logic. Set the size of the canvas so that they are equal to those of the
image, then save the image in the global variableimage(sets the value color to null) and calls the canvas.paint(null)
It also contains a method tile(final CanvasFrame [] frames), which should not be called by the event dispatch thread (EDT): it shouldn't crash, but it could slow down a lot.