Does the IP Adapter support mounting multiple IP Adapter models simultaneously and using multiple reference images at the same time?

[cjt222]

[sayakpaul]

Are you referring to something like so?
https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/adapter#combining-multiple-adapters

I don't think we support the loading of multiple IP adapters at the moment. Cc: @yiyixuxu

Do you have any convincing results from such a pipeline? Otherwise, it'd be hard for us to prioritize it.

[asomoza]

I would like to comment here since is something I'm currently looking at, multiple IP Adapters its useless if it doesn't support attention masking, but with that its really good, I don't have examples but you can watch the video of the developer that made the node for comfyui:

https://www.youtube.com/watch?v=vqG1VXKteQg&t=470s

this really opens up a lot of options for generation, also its good to use them when the images are big and you don't want to lose details, so you can divide the image and assign ip adapters to the portions of the image to make the final generation.

Finally its not that important to me since it can be done outside of diffusers, but the multiple images option is really popular right now and its better know as "instant-lora", where you feed multiple images to one IP-Adapter and they're combined in the attention layer.

[sayakpaul]

Thanks for the pointers. Would be great to have some code references if you have any.

[whiterose199187]

hi @sayakpaul

Here are some relevant discussions I found:

tencent-ailab/IP-Adapter#45
cubiq/ComfyUI_IPAdapter_plus#145 (comment)

[asomoza]

I have a couple of things to do before I can get into this but here's the invokeai implementation of the multiple adapters:

https://github.com/invoke-ai/InvokeAI/pull/4818/files

and here's the code for the ip-adapter node attention masks:

cubiq/ComfyUI_IPAdapter_plus@ebd946f

[asomoza]

I did a test in comfyui to get a better understanding:

Image 1	Image 2	Mask	Image 3

Instant lora

Image 1 and image 2 at same weight

More weight image 1

More weight image 2

All three images same weight

Multiple IP Adapters with attention masking

Using the mask for the people only (in comfyui you can assign a mask to a color) and the prompt "two women holding each other"

same weight for each adapter produces a "two faces" effect:

lowering the weight of each ip adapter produces the desired effect:

and using the third image as a background:

hope it helps to better understand how do they work together.

[whiterose199187]

Hello folks,

Is there any plan to support this capability in diffusers?

[patrickvonplaten]

cc @yiyixuxu

[yiyixuxu]

I think code wise it's pretty straightforward to support multiple ip-adapters.

However, I'm trying to understand if it makes sense to support this for every single pipeline? i.e. text2img, img2img, inpaint, controlnet?

@asomoza said it does not work great without the mask - this makes me think maybe we just need one community text2img pipeline that supports multiple ip-adapter, along with the mask. Let me know what you think!

[thibaudart]

I think code wise it's pretty straightforward to support multiple ip-adapters.

However, I'm trying to understand if it makes sense to support this for every single pipeline? i.e. text2img, img2img, inpaint, controlnet?

@asomoza said it does not work great without the mask - this makes me think maybe we just need one community text2img pipeline that supports multiple ip-adapter, along with the mask. Let me know what you think!

yes, often when we worked on some image we used:

• ipadater for style (with multiple reference images)
• another one for face
• some controlnet too

we generate a first image then we work on it using img2img and inpainting.

[asomoza]

Multiple IP Adapters without masks won't do any harm though, but IMO is just the more or less the same as one with multiple images (weighted).

I always though that diffusers pipelines were just examples so a community pipeline as an example would be sufficient and people can use it as a reference for their own. But personally I use them with controlnet and/or t2i adapters and masks all the time and almost never use them alone.

The same as a mask, I did an example with just one mask but it would be better to be able to provide each adapter with its own mask.

Probably the list for use them in a community pipeline that would be useful is

• Multiple IP Adapters with masks
• Multiple weighted images with each adapter
• Negative noise for each image (it really makes a difference)
• Controlnet and T2I Adapters
• Start and end in steps or % for each adapter

I don't see that much use for them in img2img or inpainting but I must admit I haven't tested them that much for those tasks.

[thibaudart]

no multi ip adapters without mask are so helpful:
one for style
one for face (using a different checkpoint)

[asomoza]

no worries, its just my experience, but in my tests the face adapter also interferes with the style so for me its better to use a mask for the face too.