[Pipeline] PIA (Personalized Image Animator)

[a-r-r-o-w]

In #6328, me and @jon-chuang have been looking into adding support for Image-to-Video with AnimateDiff. What we have, at the moment, is a hacky idea that has been used in webui implementations. The results obtained aren't that great but somewhat decent. In the search for improving quality, I came across PIA which has far superior quality for the image-to-video task.

Given an elaborated image generated by a personalized text-to-image model, the proposed PIA animates it with realistic motions according to different text prompts while preserving the original distinct styles and high-fidelity details.

This is amazing work and your paper was very intuitive ❤️

Code: https://github.com/open-mmlab/PIA
Paper: https://arxiv.org/abs/2312.13964
Project Page: https://pi-animator.github.io/
Model: https://huggingface.co/Leoxing/PIA/
Replicate: https://replicate.com/open-mmlab/pia
Authors: @ymzhang0319 @LeoXing1996 @zengyh1900 @fangyouqing @hellock

@DN6 @sayakpaul

[sayakpaul]

Hmm, I will let @DN6 comment here. An animator like this might indeed be quite cool. But AnimAteDiff, in its default, generates nice animations already. So, what does this one specifically help with?

[a-r-r-o-w]

There are a few things here:

• AnimateDiff does not work well with initial image input conditioning. There are hacky ways of doing it, as found by the community and as used in my PR for img2vid, but the quality isn't that competent. This approach introduces a new approach that works great with image condition.
• Here, the authors introduce a new model that replaces the unet.conv_in layer and adds a temporal alignment layer. Link to model.
• Higher motion control with prompt, probably due to how the additional/new layers are trained.
• Allows style transfer easily, which AnimateDiff vid2vid does decently on as well, and you just need an initial image.

[DN6]

@a-r-r-o-w I think I can take a stab at this one. We can see if can reach out to the authors to host the checkpoints in diffusers format as well.

[a-r-r-o-w]

@a-r-r-o-w I think I can take a stab at this one. We can see if can reach out to the authors to host the checkpoints in diffusers format as well.

Sounds great! If I can be of help, do let me know :)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.