Jaskirat Singh, Liang Zheng, Cameron Smith, Jose Echevarria
Controllable image synthesis with user scribbles is a topic of keen interest in the computer vision community. In this paper, for the first time we study the problem of photorealistic image synthesis from incomplete and primitive human paintings. In particular, we propose a novel approach paint2pix, which learns to predict (and adapt) “what a user wants to draw” from rudimentary brushstroke inputs, by learning a mapping from the manifold of incomplete human paintings to their realistic renderings. When used in conjunction with recent works in autonomous painting agents, we show that paint2pix can be used for progressive image synthesis from scratch. During this process, paint2pix allows a novice user to progressively synthesize the desired image output, while requiring just few coarse user scribbles to accurately steer the trajectory of the synthesis process. Furthermore, we find that our approach also forms a surprisingly convenient approach for real image editing, and allows the user to perform a diverse range of custom fine-grained edits through the addition of only a few well-placed brushstrokes.
We propose paint2pix which helps the user directly express his/her ideas in visual form by learning to predict user-intention from a few rudimentary brushstrokes. The proposed approach can be used for (a) synthesizing a desired image output directly from scratch
wherein it allows the user to control the overall synthesis trajectory using just few coarse brushstrokes (blue arrows) at key points,
or, (b) performing a diverse range of custom edits directly on real image inputs.
Official implementation of our Paint2pix paper with streamlit demo. By using autonomous painting agents as a proxy for the human painting process, Paint2pix learns to predict user-intention ("what a user wants to draw") from fairly rudimentary paintings and user-scribbles.
- (19/08/22) Our project demo is online. Try generating amazing artwork or realistic image media right from your browser!
real-image-editing-v1.mp4
Table of contents generated with markdown-toc
- Linux or macOS
- NVIDIA GPU + CUDA CuDNN (CPU may be possible with some modifications, but is not inherently supported)
- Python 3
- Tested on Ubuntu 20.04, Nvidia RTX 3090 and CUDA 11.5
- Dependencies:
We recommend running this repository using Anaconda. All dependencies for defining the environment are provided inenvironment/paint2pix_env.yaml.
Please download the following pretrained models essential for running the provided demo.
| Path | Description |
|---|---|
| Canvas Encoder - ReStyle | Paint2pix Canvas Encoder trained with a ReStyle architecture. |
| Identity Encoder - ReStyle | Paint2pix Identity Encoder trained with a ReStyle architecture. |
| StyleGAN - Watercolor Painting | StyleGAN decoder network trained to generate watercolor paintings. Used for artistic content generation with paint2pix. |
| IR-SE50 Model | Pretrained IR-SE50 model taken from TreB1eN for use in ID loss and id-encoder training. |
Please download and save the above models to the directory pretrained_models.
We provide a streamlit-drawble canvas based demo for trying out different features of the Paint2pix model. To start the demo use,
CUDA_VISIBLE_DEVICES=2 streamlit run demo.py --server.port 6009
The demo can then be accessed on the local machine or ssh client via localhost.
The demo has been divided into 3 convenient sections:
- Real Image Editing: Allows the user to edit real images using coarse user scribbles
- Progressive Image Synthesis: Start from an empty canvas and design your desired image output using just coarse scribbles.
- Artistic Content Generation: Unleash your inner artist! create highly artistic portraits using just coarse scribbles.
Paint2pix for progressive image synthesis
Paint2pix for achieving diverse custom real-image edits
Paint2pix for generating highly artistic content using coarse scribbles
This code borrows heavily from pixel2style2pixel, encoder4editing and restyle-encoder.
If you use this code for your research, please cite the following works:
@inproceedings{singh2022paint2pix,
title={Paint2Pix: Interactive Painting based Progressive
Image Synthesis and Editing},
author={Singh, Jaskirat and Zheng, Liang and Smith, Cameron and Echevarria, Jose},
booktitle={European conference on computer vision},
year={2022},
organization={Springer}
}
@inproceedings{singh2022intelli,
title={Intelli-Paint: Towards Developing Human-like Painting Agents},
author={Singh, Jaskirat and Smith, Cameron and Echevarria, Jose and Zheng, Liang},
booktitle={European conference on computer vision},
year={2022},
organization={Springer}
}