Video Demoireing with Relation-Based Temporal Consistency

Video Demoireing with Relation-Based Temporal Consistency (CVPR 2022)
Peng Dai, Xin Yu, Lan Ma, Baoheng Zhang, Jia Li, Wenbo Li, Jiajun Shen, Xiaojuan Qi.
Paper, Project_page, Video

Introduction

Our video demoireing network mainly consists of three parts: First, the PCD takes consecutive frames as inputs to implicitly align frames in the feature space. Second, the feature aggregation module merges aligned frame features at different scales by predicting blending weights. Third, the merged features are sent to the demoire model with dense connections to realize moire artifacts removal.

Dataset

We construct the first video demoireing dataset, containing 290 videos (720p, 60 frames for each video), and the equipment we adopted are 'TCL20 pro mobile phone + Huipu v270 monitor' and 'iPhoneXR + MackBook Pro'.

In data_v1, image pairs are aligned using the homography. In data_v2, aligned image pairs are refined using the estimated optical flow. Besides, we also provide the original moire videos. Alternatively, you can use this link. The Baidu_disk_link, and the password: b28v

Please download and unzip the dataset you need, then change the data paths in 'config/video_demoire_temporal_mbr_scratch_v2.yaml' lines 7 and 8. Note that the results in our paper are based on the data_v1.


To train a model better suited for your use or device, you can follow the 'dataset_prepare/data_prepare.py' step by step to prepare your own training data.

Environments

Create the conda virtual environment (python 3.7) and activate it.
conda create -n vdm python=3.7
conda activate vdm

Install pytorch 1.6.0 and other libraries in the 'requirements.txt'.
conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.2 -c pytorch
pip install -r requirements.txt

Install the DCNv2.
cd model/DCNv2
python3 setup.py build develop

Our implementation has been tested on one NVIDIA 2080Ti GPU with cuda 10.2.

Train

To train the model, just run:
python train.py --config config/video_demoire_temporal_mbr_scratch_v2.yaml

Note: if too large offsets appear while training the PCD model, please restart training. Alternatively, you can use this model (train two epochs) to initialize the network.

Test

To test the model, just run:
python test.py --config config/video_demoire_temporal_mbr_scratch_v2.yaml

We also provide pre-trained models for both TCL and iPhone. The 'checkpoint_000049.tar' means no relation-based loss, and 'checkpoint_000059.tar' means relation-based loss. Please download these models into the corresponding folder (line 29) and change the test epoch (line 54) as described in 'config/video_demoire_temporal_mbr_scratch_v2.yaml'.

Results

Contact

If you have any questions, you can email me ([email protected]).

Citation

If you are interested in our code or method, please consider citing the following:

@inproceedings{dai2022video,
  title={Video Demoireing with Relation-Based Temporal Consistency},
  author={Dai, Peng and Yu, Xin and Ma, Lan and Zhang, Baoheng and Li, Jia and Li, Wenbo and Shen, Jiajun and Qi, Xiaojuan},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2022}
}