Skip to content

[IP-Adapter] Adding IP-adapter masking feature #6802

@yiyixuxu

Description

@yiyixuxu

We just added multi-adapter and multi-image support for IP-Adapter #6573

Anyone interested in adding an IP adapter masking feature? It will allow you to divide the image and assign different ip-adapter to the specified portion of the image.

more information on the feature

notes on the design

  • We can add community pipelines (text-to-image and ControlNet) that use multi-ip-adapter with mask
  • or, we can integrate this feature into core diffusers if we can add this feature without adding additional code to our pipelines. Some thoughts on how to achieve this:
    1. we can add utility functions to pre-process IP-adapter mask to our image processor class
      class VaeImageProcessor(ConfigMixin):
    2. the pre-processed masked can be passed down to IPAdapterAttnProcessor/IPAdapterAttnProcessor2.0 via cross_attention_kwargs
    3. modify the IPAdapterAttnProcessor/IPAdapterAttnProcessor2.0 so it can handle mask

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions