[ip-adapter]fix IP-adapter support for SAG, panorama pipeline

[yiyixuxu]

We have added a IPAdapterTexterMixin #6862. However, we have discovered that the IP adapter support was not implemented correctly in the following pipelines:

• StableDiffusionSAGPipeline
• StableDiffusionPanoramaPipeline

If anyone is interested in fixing the IP-adapter support for these pipelines, please feel free to give it a try!

cc @a-r-r-o-w here, let me know if I missed any pipeline here

[a-r-r-o-w]

@yiyixuxu Thanks! I've already updated the LCM pipelines in 6862 :) I haven't found time to look thoroughly but there may be other pipelines that do not have the latest implementation. The IPAdapterTesterMixin is implemented with the assumption that a pipeline that inherits from IPAdapterMixin supports the ip_adapter_image_embeds parameter in __call__ and correctly initializes added_cond_kwargs like so:

added_cond_kwargs = (
    {"image_embeds": image_embeds}
    if ip_adapter_image is not None or ip_adapter_image_embeds is not None
    else None
)

# as opposed to: (which causes the tests to fail)
added_cond_kwargs = {"image_embeds": image_embeds} if ip_adapter_image is not None else None

Maybe others from the community can take a better look and update pipelines that do not have the latest implementation as introduced in #6868.

[stephen-iezzi]

I'll go for the SAG pipeline! Or if you want them completed together I can go for it in one PR

[tolgacangoz]

If not, I can go for StableDiffusionPanoramaPipeline.

[yiyixuxu]

thanks! they don't have to be completed together. I think one PR for each pipeline is better

[stephen-iezzi]

I've got the SAG pipeline pretty much completed. However, I'd like to get input on how y'all prefer I handle ip_adapter_image_embeds being passed in __call__. The SAG pipeline is the only one where there is a added_uncond_kwargs variable that goes along with added_cond_kwargs.

This is how it is set up currently:

added_cond_kwargs = {"image_embeds": image_embeds} if ip_adapter_image is not None else None
added_uncond_kwargs = {"image_embeds": negative_image_embeds} if ip_adapter_image is not None else None

added_uncond_kwargs is set directly using negative_image_embeds but with the suggested change of being able to pass ip_adapter_image_embeds in __call__ how would you like to set added_uncond_kwargs?

I couldn't find many examples of passing ip_adapter_image_embeds to see if there is an obvious way to pull the negative_image_embeds from that. Let me know what you think!

[yiyixuxu]

@Stepheni12
hi! thanks for working on SAG

I'm still working on this PR #7016 so maybe wait until after the PR merged (soon I promise) if you need to use ip_adapter_image_embeds argument

[stephen-iezzi]

@yiyixuxu I just saw that you merged #7016. The issue that I am having with completing the SAG pipeline is that the added_uncond_kwargs variable inside __call__ requires negative_image_embeds. However, with the new setup of each of the pipelines the negative_image_embeds variable doesn't exist within the call function.

In that example code you posted in the #7016 PR you are passing the ip_adapter_image_embeds variable inside your pipeline definition as ip_adapter_image_embeds=[torch.cat([negative_image_embeds, image_embeds], dim=0)],. Since negative_image_embeds is passed in within the torch.cat I can't directly access it. Now my question is, how would you like me to access the negative_image_embeds value to set added_uncond_kwargs in the SAG pipeline?

[yiyixuxu]

@Stepheni12
I see .... you could just do

ip_adapter_image_embeds = self.prepare_ip_adapter_image_embeds(...)
image_embeds = []
negative_image_embeds = []
for single_image_embeds in ip_adapter_image_embeds:
        single_image_embeds, single_negative_image_embeds = single_image_embeds.chunk(2)
        image_embeds.append(single_image_embeds)
        negative_image_embeds.append(single_negative_image_embeds)