Fix textual inversion SDXL and add support for 2nd text encoder

[dsocek]

Fix Textual Inversion SDXL fine-tuning and add support for training 2nd text encoder

Textual inversion for SDXL fine tuning script is not working as no guidance from new token is resembled in generated images.

Training Set ./cat (6 images):

Results Before Fix:

Training:

accelerate launch textual_inversion_sdxl.py \
  --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" \
  --train_data_dir="./cat" \
  --learnable_property="object" \
  --placeholder_token="<cat-toy>" \
  --initializer_token="toy" \
  --mixed_precision="bf16" \
  --resolution=768 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --max_train_steps=500 \
  --learning_rate=5.0e-04 \
  --scale_lr \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --save_as_full_pipeline \
  --output_dir="./textual_inversion_cat_sdxl"

Inference:

from diffusers import StableDiffusionXLPipeline
import torch

model_id = "./textual_inversion_cat_sdxl"
pipe = StableDiffusionXLPipeline.from_pretrained(model_id,torch_dtype=torch.float16).to("cuda")

prompt = "A <cat-toy> backpack"

image = pipe(prompt, num_inference_steps=50, guidance_scale=7.5).images[0]
image.save("cat-backpack.png")

Output (from 4 inferences):

These results show no or very poor guidance from the object token.

Results After Fix:

Same Training command

Same Inference script Output (4 inferences):

Good guidance is shown after fix (and this is after only 500 training/fine-tuning steps)

Also, now we also infer with the 2nd text encoder:

from diffusers import StableDiffusionXLPipeline
import torch

model_id = "./textual_inversion_cat_sdxl"
pipe = StableDiffusionXLPipeline.from_pretrained(model_id,torch_dtype=torch.float16).to("cuda")

prompt = "A <cat-toy> backpack"

image = pipe(prompt="", prompt_2=prompt, num_inference_steps=50, guidance_scale=7.5).images[0]
image.save("cat-backpack-prompt_2.png")

Output with 2nd text encoder from 4 inferences:

This PR also updates documentation with inference examples (see README_sdxl.md)

[dsocek]

@patrickvonplaten could kindly help assign appropriate reviewers if bandwidth is available?

[dsocek]

cc: @sayakpaul @yiyixuxu

[sayakpaul]

Thanks very much for the fix. Will merge as soon as the CI is green.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[dsocek]

@sayakpaul Thanks, I seem to forgot to run make style/quality, just added a style fix commit

[sayakpaul]

Thanks for your awesome contributions!

[dsocek]

@sayakpaul Thank you very much for taking time to review!