Releases

libxcam 1.5.0 release

2020/12/24: release libxcam version 1.5.0

• 360 stitching GLES pipeline supports processing YUV420p pixel format

libxcam 1.4.0 release

2020/06/01: release libxcam version 1.4.0

• 360-degree video stitching supports processing YUV420p pixel format
• 360-degree video stitching quality tuning
  • support json parse to apply OpenCV fisheye camera calibration parameters
• 8K 360-degree video stitching performance improvement
  • optimized AVX pixel data loading instructions
• Implement new FFmpeg xcam video filter
  • extend capis to expose video stitching functionality
  • support soft/GLES/Vulkan image stitching
• Misc
  • simplify configure.ac
  • extract macros to m4 file
  • remove Beignet dependence
  • limit OpenCV version to [3.0.0, 4.0.0)

libxcam 1.3.0 release

2019/09/11: release libxcam version 1.3.0

• 8K 360 degree image stitching performance improvement
  • enable AVX512 intrinsic instructions to increase performance
• 3 cameras 8K stitching enabling (CPU/GLES)
  • support disable feature match in runtime
  • support specify the pyramid levels of blender
  • stitching quality tuning
• 2 cameras 1080p stitching enabling (CPU/GLES)
• DNN inference support processing video stream
• Use Intel open source graphics compute runtime for OpenCL (NEO)
  • will not support Beignet driver

libxcam 1.2.2 release

2019/04/02: release libxcam version 1.2.2

• DNN inference framwork enabling.
  • enable pedestrian and vehicle detection based on OpenVino
• OpenCV feature match enabled for image stitching pipeline (GLES/Vulkan/CPU)
  • support three versions of feature match: default, cluster, capi
• Abstract OpenCV source in ocv module, and build into a separate static library

libxcam 1.2.1 release

2018/12/19: release libxcam version 1.2.1

• Vulkan surround-view stitching enabling.
  • enable features of multi-band blender and geometry remap.
  • enable Vulkan image stitching. (feature-match may support later)
• Surrond-view texture rendering.
  • enable 3D surround view scene rendering and manipulation.
• require OpenSceneGraph minimal version 3.3.2

libxcam 1.2.0 release

2018/10/16: release libxcam version 1.2.0

• GLES compute framework enabling.
  • enabled GL image processing framework.
  • enable features of multi-band blender, geometry remap and data-copy.
• GLES surround-view stitching enabling.
  • GLES stitching enabled (feature-match may support later).
  • performance optimized on compute shaders.
• Vulkan compute framework enabling.
  • enabled Vulkan image processing under xcam framework.
  • vk-copy-handler as sample code.
• suggest OpenCV version [3.0.0 - 3.4.3]
• support CPU stitching based on dual-const and dual-curve scaling factors.
• support OpenCL stitching based on auto-scale in vertical 2D-direction.
• change test-soft-image to test-surround-view

libxcam 1.1.0 release

2018/03/01: release libxcam version 1.1.0

• CPU stitching enabled for automotive surround view in Linux/Android.
  • enable CPU multi-thread image processing framework.
  • enable multi-band blender, geometry remap, data-copy, OpenCV feature match.
  • enable 3D-bowl model stitching.
  • support generic Android platform.
  • CPU version of surround view stitching upstream to Android Open Source Project.
• surround-view OpenCL stitching feature enabled and quality improvement.
  • add dewarp process based on bowl view model in geometry map.
  • support multiple (4) cameras stitching.
  • add new selection method based on clustering in feature match to improve quality of stitching.
  • auto scale in x direction is done and y direction is WIP.
  • quality tune on different datasets, both indoor and outdoor.
• support standard OpenCL 2.0+ driver (VPG OCL driver)
  • support standard OpenCL buffer.
  • remove libdrm dependency.
  • drm preview removed from test-device-manager, test-pipe-manager and usb camera.
• enable deblurring feature based on OpenCV. (GSoC program)
  • based on deconvolution algorithm filters
  • support noise estimation
  • support edgetaper
• sourround view solution will be integrated into Android EVS (exterior view system) WIP
• add sample recipe for yocto build (libxcam/doc/yocto).

libxcam 1.0.0 release

2017/07/10: release libxcam version 1.0.0

• 360 video stitching performance and quality improvement.
  • enable geometry map to improve performance.
  • quality tuned on different resolutions (1080P and 4K).
  • support CPU and OpenCL path in feature match.
  • enable lens shading correction based on fisheye image.
• gyroscope-based video stabilization enabling.
  • enable gyroscope 3-DoF (orientation) assist video stabilization.
  • orientation (gyro) data should be measured by quaternion as the pose of target frame reference to base frame
• CL framework refine.
  • enable CL argument template instead of member variables in kernel.
  • CL kernel support async mode.
  • image handler take over input/output buffer management from image kernel.
• prepare libxcam debian package and fix most warnings.

libxcam 0.9.0 release

2017/04/01: release libxcam version 0.9.0

• 360 video stitching improvement.
  • remove sharp boundary in sphere mode by local/global area resize.
  • tune algorithm parameters to preserve blinking.
  • integrate 360 stitching pipeline into libxcam module.
  • gst-xcam-filter supports 360 stitching case.
• export C APIs for each OpenCL feature.
  • support stich/3d-nr/wavelet-nr/defog features by xcam handle.
  • be designed for FFmpeg xcam video filter usage.

libxcam 0.8.0 release

2016/12/30: release libxcam version 0.8.0

• Digital Video Stabilization (DVS)
  • enable DVS smart analysis plugin.
  • algorithms used in DVS (feature detect, optical flow, motion model estimation) are based on OpenCV.
  • implement image warp perspective algorithm in CL kernel.
• Image stitching to create panorama photography
  • implement fisheye calculation which map location to equirectangular.
  • enable feature detection and match based on OpenCV optical flow algorithm.
  • enable seam mask to improve the quality of blender.
• gstreamer plugin xcamfilter can co-work with xcamsrc

libxcam 0.7.0 release

2016/09/30: release libxcam version 0.7.0

• 3D-NR quality and performance improvement.
  • use image exposure parameters to adjust global denoise strength.
  • calculate image local variation to adjust local denoise strength.
  • optimize OCL kernel by shared local memory and sub-group to improve performance.
• defog/dehaze quality improvement based on DCP(dark channel prior).
  • enable dark channel prior to improve hazy image quality.
  • fix image corruption in recover step.
  • add lateral-range filter to smooth dark channel and remove halo effect.
• enable multi-band blender for image stitching.
  • based on gaussian and laplacian pyramid and blend images in each band, then reconstruct stitching image from pyramid level.
  • blender can smooth seams and prevent ghosting.
• enable geometry correction.
  • map 2D image from src to dst image based on coordinate table which is center aligned.
  • prevent sawtooth edge by OCL sampler.
• gst-xcamfilter feature supports.
  • xcamfilter can co-work with different camera source.

libxcam 0.6.1 release

2016/07/29: release libxcam version 0.6.1

• enable 3D noise reduction, performance stay tuned.
  • suppress image noise in both temporal and spatial domain.
  • the algorithm applies denoise filter based on pixel-block differences in neighbor blocks and previous frames.
  • the processed image is much more clean and preserves motion objects edge in low-light conditions.
• support gst xcamfilter plugin which is independent from camera source.
  • xcamfilter plugin can work with gstreamer source plugins, e.g. videotestsrc/filesrc/v4l2src.
  • xcamfilter features support defog/wireframe/waveletNR/3d-denoise.
  • sink pad supports format NV12, src pad supports DMA buffer-sharing for performace if links to HW plugin(encoder).
  • wireframe/scaler/waveletNR moved to post image processor which can co-work with isp processor(as pre-image processor).

libxcam 0.6.0 release

2016/06/30: release libxcam version 0.6.0

• enable block-split version of histogram based WDR tone-mapping algorithm.
  • divide image into several blocks and apply bidirectional log based function on luminance to improve the contrast inside each block.
  • apply weight function based on distance between target pixel and center pixel of each neighbor block to smooth and eliminate block boundaries.
  • apply weight function based on pixel intensity deviation between neighbor blocks keep differentiation and weaken boundaries.
  • image quality is improved and more details in both over-exposured and under-exposured area can be shown clearly on screen.
• Bayesian wavelet shrinkage for adaptive noise filtering.
  • enable adaptive noise filtering based on Bayesian shrinkage estimation
  • estimate the image’s background noise level by analyzing. wavelet coefficient.
  • optimize the threshold T which minimizes the Bayesian risk, i.e, the expected value of the mean square error.
• Haar wavelet NR tuning improvement.
  • fine tuning the threshold statistic data for ultra-low light condition. to prevent edge from over smoothing.
• face detection supports.
  • enable face detection framework and support FD plugins.
  • enable wire-frame to track faces.
• fog removal improvement based on Retinex algorithm.
  • support multi-scale algorithm to reduce halo effects.
  • tune parameters to keep brightness and clear edge.
  • improve saturation based on chroma enhancement.
• image processing service framework.
  • add pipe-manager to support smart-analysis and post image processor.
  • separate major code into different modules, e.g. xcore/isp/ocl/3a.
  • support NV12 stream as fake input instead of camera sensor.

libxcam 0.5.0 release

2016/03/28: release libxcam version 0.5.0

• enable new WDR tone mapping based on histogram adjustment.
  • apply bidirectional log based function on luminance to improve image contrast.
  • transmit wide dynamic range image data to 8-bit color data by novel histogram adjustment algorithm.
  • details in both over-exposured and under-exposured area clearly show up on screen.
• enable wavelet-based algorithm for noise reduction.
  • decompose image into multiple-scales by wavelet transform with low pass and high pass filters.
  • perform soft threshold on smaller coefficients(high-pass) to reduce noise.
  • threshold should be suitably decided by tuned parameters.
  • reconstruct image from scaled low-pass and high-pass images.
  • enable Haar-wavelet and Hat-wavelet NR filters.
• enable fog removal feature based on retinex algorithm.
  • base on retinex single-scale algorithm.
  • scale down image for gaussian blur to improve performance.
  • scale up blured image and amplify log differences with original image.
• improve sharpness in normal light with extreme profile.
• support cl post processing with isp mode.
• support swap-buffer on nv12 format if only single plane need to process.

libxcam 0.4.0 release

2015/12/30: release libxcam version 0.4.0

• improve performance on OCL pipeline.
  • design new formats and pipeline to save memory IO.
  • improve bayer noise reduction with bilateral filter.
  • support edge enhancement into bayer pipe.
  • make gpu/cpu working together for 3a-statistics calculation.
  • move tonemapping kernel from RBG to bayer format.
• improve WDR-tonemapping algorithm for quality.
  • utilize an new adaptive local tone mapping algorithm.
  • provide the mapping between 16-bit wide dynamic range images to 8-bit images to have “nice looking”.
  • preserve more information on luminance values of the scene especially for high contrast images.
• tonemapping parameters calculated to increase the contrast between high light and low light.
  • tone mapping regularization parameters are calculated according to the distribution of brightness histogram.
  • the local adaptive parameter are calculated by weighting the neighbor pixels with a Gaussian blur filter.
• add bayer raw video input to simulate camera sensor.

libxcam 0.2.1 release

libxcam 0.2.1 mainly provided some performance tuning fixes and ability to work with intel's new 3A library.

libxcam 0.2.1 release link: libxcam 0.2.1 release

0.2.1 detailed features:

libxcam 0.2.1 provided combined cl kernel for basic pipeline(blc, 3a-stats, wb, gamma, demosaic, macc, csc) which can save the IO bandwidth.

libxcam 0.2.1 provided the async processing and tiling surface operation.

libxcam 0.2.1 can work with intel's new 3A library and firmware.

libxcam 0.2.0 release

libxcam 0.2.0 provided the basic and advanced features for image capturing and processing by opencl.

0.2.0 release branch: github 0.2.0 branch

0.2.0 detailed features:

Basic pipeline

• Statistic info extraction

It defines standard statistics info APIs for external users.

• Customized 3A reference code

It’s a simple analyzer as reference code of how to implement a 3A analyzer based on libXCam core(C plusplus version).

In the future, this will be moved to 3alib based on XCam3ADescription(C APIs) as reference for a simple dynamical 3a lib.

Customer could reference this code to provide their own 3A lib, but it must provide a full 3A analysis.

• Demosic

Demosaic Raw bayer format into any collection of RGBA/RGBx (8bit/16bit) pixel image.

• Black level adjustment

This module includes 2 parts: Black Level Correction and normalization. The coming image data no matter what format it is, do the black level correction first, and then transform to 16bit.

• White balance

It’s in bayer layer operation. Every pixel’s Gr,B, R,Gb’s value will be multiplied the gain. It’s mainly used for white balance adjustment.

• tone curve and gamma

It’s gamma adjustment, to change the pixel’s RGB value to the new RGB value according to gamma table.

• Multi Axis Color correction-MACC

It divided UV axis to 16 different UV areas. According to this pixel’s UV value, choose the corresponding UV area’s macc table coefficient, and calculate the new UV value.

• Color effect applying and color space conversion

Color effect applying uses color matrix to support rgb to yuv and color effect. It’s use for the coefficient adjustment when converting RGB to NV12.

Color space conversion provides the different color space conversion. It includes rgb2nv12，rgb2lab，rgb64torgb32，nv12torgb，yuyv2rgb。

• local preview

It uses libdrm’s kms to preview the video data after libxcam’s operations. It uses double buffer and primary plane to show the video data. We tried to use overlay to show the YUYV data directly, but some unstable phenomenon showed up, so we used primary plane to do it.

• gst xcamsrc plugin

Gst xcamsrc plugin consists of two parts: xcam 3a interface (libgstxcaminterface.so) and xcam gstreamer plugin (libgstxcamsrc.so).

a. xcam 3a interface provides external APIs to configure manual AE/AWB settings, various effects, and specific CL processing features (HDR/denoise/gamma). xcam gstreamer plugin provides the common plugin functionalities and talks to libxcam framework via DeviceManager and X3aAnalyzer.

b. libxcam pipeline’s Image processor and 3a analyzer can be configured by setting properties to xcam gstreamer plugin. E.g. application can set ISP_IMAGE_PROCESSOR + AIQ_ANALYZER to build legacy pipeline similar to v4l2src, and CL_IMAGE_PROCESSOR + DYNAMIC_ANALYZER to build pipeline with GPU accelerated image processing and dynamic loaded 3A algorithms.

Advanced Pipeline

• Defect Pixel Correction-DPC DPC is performed on bayer data for defect pixel detection and correction. 4 parameters are provided to adjust DPC: gr_threshold: GR threshold of defect pixel correction

  r_threshold: R threshold of defect pixel correction

  b_threshold: B threshold of defect pixel correction

  gb_threshold: GB threshold of defect pixel correction

• Temporal Noise Reduction (TNR)

Temporal video denoising method provides the best image quality especially in low light. In this method, noise between frames is reduced.

• YEENR

It’s based on Y plane of YUV color space. It operates noise reduction and edge enhancement.

• SNR

Simple Noise Reduction is using the average filtering refer to the 9 pixels nearby, including the pixel itself. And it is performed on RGBA color space.

• Bi-lateral Noise Reduction

Based on “Bilateral Filtering for Gray and Color Images”, we implemented a cl version. Bilateral NR makes images smooth while preserving edges, by means of a nonlinear combination of nearby image values.

• HDR on LAB for 10 bits

Realize high dynamic range on Lab color space. Use RGB to get Lab value and do high dynamic on it, then translate Lab back to RGB. And it turns out to have a better quality on Lab than RGB.