IP-Adapter Face Id

[fabiorigano]

Model/Pipeline/Scheduler description

A new IP-Adapter Face model has been released. It uses image embeddings from a face recognition model instead of CLIP image embedding. In addition to this, it uses LoRa to improve ID consistency.

Do you plan to support this model? I am happy to contribute

Open source status

• The model implementation is available.
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

https://github.com/tencent-ailab/IP-Adapter
https://huggingface.co/h94/IP-Adapter-FaceID

[fabiorigano]

Here some results using the same prompt of IP Adapter Full:

Code

# Generate image embedding with thirdy party service
app = FaceAnalysis(name="buffalo_l", providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
app.prepare(ctx_id=0, det_size=(640, 640))

image = cv2.imread("person.jpg")
faces = app.get(image)

faceid_embeds = torch.from_numpy(faces[0].normed_embedding).unsqueeze(0)

# Load IP Adapter 
from ip_adapter.ip_adapter_faceid import IPAdapterFaceID
ip_model = IPAdapterFaceID(pipe, ip_ckpt, "cuda")

# Inference
negative_prompt = "monochrome, lowres, bad anatomy, worst quality, low quality, blurry"
prompt="A photo of a girl wearing a black dress, holding red roses in hand, upper body, behind is the Eiffel Tower"
images = ip_model.generate(
    faceid_embeds=faceid_embeds, num_samples=4, prompt=prompt,
    width=512, height=704, num_inference_steps=50, seed=0, negative_prompt=negative_prompt)
grid = image_grid(images, 1, 4)
grid

Input image

Output