regression - vae load sets inccorect scaling factor

[vladmandic]

Describe the bug

code added via this pr (refactor update from single file)
forces scaling factor to fixed value regardless of what is being loaded

code reference:

newly added code in: https://github.com/huggingface/diffusers/blame/7c8cab313e4c66a813d146bcf92023b0489a2369/src/diffusers/loaders/single_file_utils.py#L1177

this results in washed out images when sdxl model is used with additionally loaded vae.

correct values should be:

• LDM VAE (sd-15) uses scaling_factor 0.18125
• SGM VAE (sd-xl) uses scaling factor 0.13025

looking at code before this pr, no scaling_factor was enforced on vae load and it worked just fine - so this is a regression in diffusers 0.26 compared to 0.25.

reference issue: vladmandic/sdnext#2824

Reproduction

n/a

Logs

No response

System Info

diffusers==0.26.2

Who can help?

@yiyixuxu @sayakpaul @DN6 @patrickvonplaten

[vladmandic]

example images with diffusers==0.25 and diffusers==0.26 with no code changes.

[DN6]

@vladmandic I believe this was fixed with #6779 and the fix should be in version 0.26.2. Would you able to test with to confirm?

[vladmandic]

its not, that PR just removes the default and leaves decision making more upstream - but if you trace it, class AutoencoderKL will always default to LDM config which means that scale_factor will be set incorrectly in config itself.
so doesn't matter if default param is removed if it then falls back to using incorrect param from config.

if you trace it, it goes via fetch_original_config and at the bottom of the stack, infer_original_config_file only handles main model, it doesn't cater to VAE - so whichever VAE is loaded, it will always use v1 config (LDM)

[DN6]

@vladmandic I'm afraid I'm not able to reproduce with the latest diffusers version.

If I try running this snippet using diffusers version 0.26.2 I get the correct VAE scaling factor for SDXL. I included external VAE loading as mentioned in the linked issue.

from diffusers import AutoencoderKL, StableDiffusionPipeline, StableDiffusionXLPipeline

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix")
pipe = StableDiffusionXLPipeline.from_single_file(
    "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors", vae=vae
)

# should be 0.13025
print(pipe.vae.config.scaling_factor)

Do you have a code snippet I could use to reproduce?

[vladmandic]

if you're loading model and with vae, config is going to be set correctly since infer_original_config_file is going to infer correct config from the model.
issue is if you're loading vae after-the-fact. and i don't want to reload entire model every time i'm switching a vae.

from diffusers import AutoencoderKL, StableDiffusionPipeline, StableDiffusionXLPipeline
pipe = StableDiffusionXLPipeline.from_single_file("https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors")
pipe.vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix")

[DN6]

@vladmandic Ran the following snippet and the scaling factor for SDXL is still 0.13025. Could you confirm that you're using diffusers 0.26.2 in your testing?

from diffusers import AutoencoderKL, StableDiffusionPipeline, StableDiffusionXLPipeline
pipe = StableDiffusionXLPipeline.from_single_file("https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors")
pipe.vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix")
print(pipe.vae.config.scaling_factor)

[vladmandic]

sorry, typo in my reproduction.
using AutoencoderKL.from_pretrained works fine since config is looked up in HF model repo.
using AutoencoderKL.from_single_file fails.

import os
from diffusers import AutoencoderKL, StableDiffusionPipeline, StableDiffusionXLPipeline
folder = '/mnt/models'
pipe = StableDiffusionXLPipeline.from_single_file(os.path.join(folder, 'stable-diffusion/base/sdxl-base-v10.safetensors'))
pipe.vae = AutoencoderKL.from_single_file(os.path.join(folder, 'VAE/sdxl-1.0.safetensors'))
print(pipe.vae.config.scaling_factor)

0.18125

[DN6]

@vladmandic Able to reproduce. Opened a PR with a fix.

[vladmandic]

closing.