-
Notifications
You must be signed in to change notification settings - Fork 6.6k
Closed
Description
We just added multi-adapter and multi-image support for IP-Adapter #6573
Anyone interested in adding an IP adapter masking feature? It will allow you to divide the image and assign different ip-adapter to the specified portion of the image.
more information on the feature
- a video about the feature in ComfyUI https://www.youtube.com/watch?v=vqG1VXKteQg&t=470s
- relevant code from ComfyUI cubiq/ComfyUI_IPAdapter_plus@ebd946f
notes on the design
- We can add community pipelines (text-to-image and ControlNet) that use multi-ip-adapter with mask
- or, we can integrate this feature into core diffusers if we can add this feature without adding additional code to our pipelines. Some thoughts on how to achieve this:
- we can add utility functions to pre-process IP-adapter mask to our image processor class
class VaeImageProcessor(ConfigMixin): - the pre-processed masked can be passed down to
IPAdapterAttnProcessor/IPAdapterAttnProcessor2.0viacross_attention_kwargs - modify the
IPAdapterAttnProcessor/IPAdapterAttnProcessor2.0so it can handle maskdef __call__(
- we can add utility functions to pre-process IP-adapter mask to our image processor class