DeepMCBM

Authors: Guy Erez, Ron Shapira Weber, and Oren Freifeld.

This code repository corresponds to our ECCV '22 paper: DeepMCBM: A Deep Moving-camera Background Model. DeepMCBM is a novel 2D-based method for unsupervised learning of a moving-camera background model, which is highly scalable and allows for relatively-free camera motion.

Table of Contents

1. Documentation
   • Environment
   • Train, Predict and Evaluate
   • Input, Output and Checkpoints
   • Predict a Pretrained Model
2. Results
3. Visual Comparisons

Documentation

Environment

The repository is equipped with a DeepMCBM_env.yml file.
Run conda env create -f DeepMCBM_env.yml from your terminal to set a conda environment using this file.
To ensure the environment is set properly, activate the new environment and run a "dry run" with few epochs:

conda activate DeepMCBM
python src/DeepMCBM.py --DryRun

Train, Predict and Evaluate

To train, predict and evaluate a deepMCBM module on the default tennis sequence:

python src/DeepMCBM.py 

Input, Output and Checkpoints

The default values for the input, output, and checkpoints paths are set in src/args.py and can be changed to any path you wish. The requirement for the input directory is to have the following subdirectories: "frames" include the sequence frames, and if ground truth labels are available, a "GT" directory containing the ground truth frames. See the input/tennis for an example. The output directories are named by the sequence and the log_name argument: output/sequence_name/log_name in this directory you will find:

• background_estimation directory containing the background estimation of the model.
• MSE directory containing the Mean Square Error (MSE) computed using the ground truth labels.
• panoramic_robust_mean.png image, shows the alignment result of the STN module.

You can change the log_name simply by adding log_name "my_new_name" to your command line.

Predict Using a Pretrained Model

To only predict and evaluate metrics:

python src/DeepMCBM.py --no_train_BMN --no_train_STN 

You can change the loaded checkpoint using a flag:

python src/DeepMCBM.py --no_train_BMN --no_train_STN --BMN_ckpt ckpt_file.ckpt  

Or by editing the MCBM_CKPT argument in src/args.py

Note: when using a pretrained model, the argument --pad, describing the size of the padding, must be the same as in the training phase.

Results

tennis.mp4

dog-gooses.mp4

flamingo.mp4

horsejump-high.mp4

stunt.mp4

zoomInZoomOut.mp4

bmx-trees.mp4

Visual Comparisons

tennis.pdf

flamingo.pdf

dog-gooses.pdf

bmx-trees.pdf

horsejump-high.pdf