Project

.gitignore

@@ -0,0 +1,4 @@
+model_logs/
+neuralgym_logs/
+data/
+models/

README.md

@@ -0,0 +1,98 @@
+# Generative Image Inpainting with Contextual Attention
+
+[CVPR 2018 Paper](https://arxiv.org/abs/1801.07892) | [ArXiv](https://arxiv.org/abs/1801.07892) | [Project](http://jhyu.me/posts/2018/01/20/generative-inpainting.html) | [Demo](http://jhyu.me/posts/2018/01/20/generative-inpainting.html#post) | [BibTex](#citing)
+
+<img src="https://user-images.githubusercontent.com/22609465/35317673-845730e4-009d-11e8-920e-62ea0a25f776.png" width="425"/> <img src="https://user-images.githubusercontent.com/22609465/35317674-846418ea-009d-11e8-90c7-652e32cef798.png" width="425"/>
+<img src="https://user-images.githubusercontent.com/22609465/35317678-848aa3fc-009d-11e8-84a5-01be01a31fc6.png" width="210"/> <img src="https://user-images.githubusercontent.com/22609465/35317679-8496ab84-009d-11e8-945c-e1f957b04288.png" width="210"/>
+<img src="https://user-images.githubusercontent.com/22609465/35347783-c5e948fe-00fb-11e8-819c-8212d4edcfd3.png" width="210"/> <img src="https://user-images.githubusercontent.com/22609465/35347784-c5f4242c-00fb-11e8-8e46-5ad224e15096.png" width="210"/>
+
+Example inpainting results of our method on images of natural scene (Places2), face (CelebA) and object (ImageNet). Missing regions are shown in white. In each pair, the left is input image and right is the direct output of our trained generative neural networks without any post-processing.
+
+## Run
+
+0. Requirements:
+    * Install python3.
+    * Install [tensorflow](https://www.tensorflow.org/install/) (tested on Release 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0).
+    * Install tensorflow toolkit [neuralgym](https://github.com/JiahuiYu/neuralgym) (run `pip install git+https://github.com/JiahuiYu/neuralgym`).
+1. Training:
+    * Prepare training images filelist.
+    * Modify [inpaint.yml](/inpaint.yml) to set DATA_FLIST, LOG_DIR, IMG_SHAPES and other parameters.
+    * Run `python train.py`.
+2. Resume training:
+    * Modify MODEL_RESTORE flag in [inpaint.yml](/inpaint.yml). E.g., MODEL_RESTORE: 20180115220926508503_places2_model.
+    * Run `python train.py`.
+3. Testing:
+    * Run `python test.py --image examples/input.png --mask examples/mask.png --output examples/output.png --checkpoint model_logs/your_model_dir`.
+
+## Pretrained models
+
+[CelebA-HQ](https://drive.google.com/open?id=1lpluFXyWDxTY6wcjixQGWX8jxUUMlyBW) | [Places2](https://drive.google.com/open?id=1M3AFy7x9DqXaI-fINSynW7FJSXYROfv-) | [CelebA](https://drive.google.com/open?id=1sP8ViF3mxUMN--xpKqonEeW9d8S8pJEo) | [ImageNet](https://drive.google.com/open?id=136APWSdPRAF7-XoS8sMBTLV-X3f-ogE0) | 
+
+Download the model dirs and put it under `model_logs/` (rename `checkpoint.txt` to `checkpoint` because google drive automatically add ext after download). Run testing or resume training as described above. All models are trained with images of resolution 256x256 and largest hole size 128x128, above which the results may be deteriorated. We provide several example test cases. Please run:
+
+```bash
+# Places2 512x680 input
+python test.py --image examples/places2/wooden_input.png --mask examples/places2/wooden_mask.png --output examples/output.png --checkpoint_dir model_logs/release_places2_256
+# CelebA 256x256 input
+python test.py --image examples/celeba/celebahr_patches_164036_input.png --mask examples/center_mask_256.png --output examples/output.png --checkpoint_dir model_logs/release_celeba_256/
+# CelebA-HQ 256x256 input
+# Please visit CelebA-HQ demo at: jhyu.me/demo
+# ImageNet 256x256 input
+python test.py --image examples/imagenet/imagenet_patches_ILSVRC2012_val_00000827_input.png --mask examples/center_mask_256.png --output examples/output.png --checkpoint_dir model_logs/release_imagenet_256
+```
+
+**Note:** Please make sure the mask file completely cover the masks in input file. You may check it with saving a new image to visualize `cv2.imwrite('new.png', img - mask)`.
+
+## TensorBoard
+
+Visualization on [TensorBoard](https://www.tensorflow.org/programmers_guide/summaries_and_tensorboard) for training and validation is supported. Run `tensorboard --logdir model_logs --port 6006` to view training progress.
+
+## License
+
+CC 4.0 Attribution-NonCommercial International
+
+The software is for educaitonal and academic research purpose only.
+
+## FAQ
+
+
+* Can other types of GANs work in current setting?
+
+The proposed contextual attention module is independent of GAN losses. We experimented with other types of GANs in Section 5.1 and WGAN-GP is used by default. Intuitively, during training, pixel-wise reconstruction loss directly regresses holes to the current ground truth image, while WGANs implicitly learn to match potentially correct images and supervise the generator with adversarial gradients. Both WGANs and reconstruction loss measure image distance with pixel-wise L1.
+
+* How to determine if G is converged?
+
+We use a slightly modified WGAN-GP for adversarial loss. WGANs are demonstrated to have more meaningful convergent curves than others, which is also confirmed in our experiments.
+
+* The split of train/test data in experiments.
+
+We use the default training/validation data split from Places2 and CelebA. For CelebA, training/validation have no identity overlap. For DTD texture dataset which has no default split, we sample 30 images for validation.
+
+* How does random mask generated?
+
+Please check out function `random_bbox` and `bbox2mask` in file [inpaint_ops.py](/inpaint_ops.py).
+
+* Parameters to handle the memory overhead.
+
+Please check out function `contextual_attention` in file [inpaint_ops.py](/inpaint_ops.py).
+
+* How to implement contextual attention?
+
+The proposed contextual attention learns where to borrow or copy feature information from known background patches to reconstruct missing patches. It is implemented with TensorFlow conv2d, extract_image_patches and conv2d_transpose API in file [inpaint_ops.py](/inpaint_ops.py). To test the implementation, one can simply apply it on RGB feature space, using image A to reconstruct image B. This special case then turns into a naive style transfer.
+
+<img src="https://user-images.githubusercontent.com/22609465/36634042-4168652a-1964-11e8-90a9-2c480b97eff7.jpg" height="150"/> <img src="https://user-images.githubusercontent.com/22609465/36634043-4178580e-1964-11e8-9ebf-69c4b6ad52a5.png" height="150"/> <img src="https://user-images.githubusercontent.com/22609465/36634040-413ee394-1964-11e8-8d23-f86a018edf01.png" height="150"/>
+
+```bash
+python inpaint_ops.py --imageA examples/style_transfer/bnw_butterfly.png  --imageB examples/style_transfer/bike.jpg --imageOut examples/style_transfer/bike_style_out.png
+```
+
+
+## Citing
+```
+@article{yu2018generative,
+  title={Generative Image Inpainting with Contextual Attention},
+  author={Yu, Jiahui and Lin, Zhe and Yang, Jimei and Shen, Xiaohui and Lu, Xin and Huang, Thomas S},
+  journal={arXiv preprint arXiv:1801.07892},
+  year={2018}
+}
+```

README1.md

@@ -0,0 +1,37 @@
+# Inpaint Object Remover
+This is the implementation of the algorithm described in the paper "Region Filling and Object Removal by
+Exemplar-Based Image Inpainting" by A. Criminisi et al.
+
+## Setup
+Requires python 3.5 or newer. You can download it [here](https://www.python.org/downloads/).
+
+### Lunix and Mac OS X
+Inside the project's directory run:
+```
+pip install -r requirements.txt
+```
+to install the dependencies.
+
+### Windows
+Download NumPy and SciPy from [here](http://www.lfd.uci.edu/~gohlke/pythonlibs/), being the number after cp your version of python. (e.g. if you have python 3.5 64 bits download **numpy‑1.13.1+mkl‑cp35‑cp35m‑win_amd64.whl** and **scipy‑0.19.1‑cp35‑cp35m‑win_amd64.whl**)
+
+Then run `pip install <path to downloaded file>` for each file.
+
+After this, inside the project's directory, run:
+```
+pip install -r requirements.txt
+```
+to install the other dependencies.
+
+## How to use
+Inside the project's directory run:
+```
+python inpainter <path to image> <path to mask>
+```
+You can also pass the `--plot-progress` argument to watch the image being created. For example, use:
+```
+python inpainter resources/image1.jpg resources/mask1.jpg --plot-progress
+```
+to process image1 inside resources folder using mask1 while printing each result. I provided some images from the paper inside the resources folder.
+
+Use `python inpainter -h` to show the available arguments.

bg/inpainter/__main__.py

@@ -0,0 +1,55 @@
+import argparse
+from skimage.io import imread, imsave
+
+from inpainter import Inpainter
+
+
+def main():
+    args = parse_args()
+
+    image = imread(args.input_image)
+    mask = imread(args.mask, as_grey=True)
+
+    output_image = Inpainter(
+        image,
+        mask,
+        patch_size=args.patch_size,
+        plot_progress=args.plot_progress
+    ).inpaint()
+    imsave(args.output, output_image, quality=100)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        '-ps',
+        '--patch-size',
+        help='the size of the patches',
+        type=int,
+        default=9
+    )
+    parser.add_argument(
+        '-o',
+        '--output',
+        help='the file path to save the output image',
+        default='output.jpg'
+    )
+    parser.add_argument(
+        '--plot-progress',
+        help='plot each generated image',
+        action='store_true',
+        default=False
+    )
+    parser.add_argument(
+        'input_image',
+        help='the image containing objects to be removed'
+    )
+    parser.add_argument(
+        'mask',
+        help='the mask of the region to be removed'
+    )
+    return parser.parse_args()
+
+
+if __name__ == '__main__':
+    main()