[Flux IPadapter] Support Xlabs IPadapter in diffusers

[chuck-ma]

It'd be great to have XLabs IPadapter supported in diffusers.

Code: https://github.com/XLabs-AI/x-flux/
Checkpoint: https://huggingface.co/XLabs-AI/flux-ip-adapter/tree/main

@sayakpaul

[sayakpaul]

Feel free to start a PR if you're interested to contribute it. We can provide guidance along the way.

Cc: @fabiorigano too

[chuck-ma]

Ok, I will have a try as long as I have time. Besides, I'm also curious if it will also work to make style transfer like we did in SDXL ipadapter.

[CONSTANT1386]

I have completed the training script and code for FLUX IP-Adapter based on diffusers, but I still encounter some issues in the final effect, Anyone can help me?

[asomoza]

Hi, you probably will have better luck asking in the original code repository, we don't even have official support for inference here yet.

[TinyTigerPan]

I have completed the training script and code for FLUX IP-Adapter based on diffusers, but I still encounter some issues in the final effect, Anyone can help me?

Hi, I'm working on FLUX IP-Adapter training script based on x-flux. And I also meet some problem, maybe we can communicate together. Here is my email [email protected]

[max-fofanov]

Hello guys, I'm also interested in collaborating on this issue! My email is [email protected].

[vladmandic]

with popularity of flux, this is becoming one of the top user requests - and hints on possible implementation?

[sayakpaul]

Cc: @asomoza

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[tariq659891]

Hello @sayakpaul,

I came across your GitHub repo and had the chance to read your Word doc—amazing work! I'm a big fan! While exploring the repo, I looked for areas where I could contribute and stumbled upon this one. I started working on it last night.

So far, I haven’t found any existing implementation of the IP adapter in Diffusers, so I decided to give it a shot. I’ve implemented it, and while it’s not producing high-quality results yet, I can see the impact of the IP adapter.

I’m sharing the code here for anyone who’s interested in checking it out or continuing the work.

Additionally, I’ve included inference code that runs on less than 17GB of memory, with plans to optimize it further to go below 14GB.

https://github.com/tariq659891/Fluxforge-inference/tree/main

[sayakpaul]

Hello @tariq659891 thanks for your kind words and also thanks for working on this.

I will let @asomoza comment on this further.