```from_single_file``` fails on certain civit checkpoints.

[Gothos]

Describe the bug

Trying to load this or this this results in a shape mismatch error with key sizes (640,640) and (640,640,1,1).

Reproduction

# First wget https://civitai.com/api/download/models/450187?type=Model&format=SafeTensor&size=pruned&fp=fp16 and rename to dreamshaper_inpaint.safetensors
from diffusers import StableDiffusionXLInpaintPipeline

StableDiffusionXLInpaintPipeline.from_single_file("dreamshaper_inpaint.safetensors")

Logs

Throws error

: Cannot load because down_blocks.1.attentions.0.proj_in.weight expected shape tensor(..., device='meta', size=(640, 640, 1, 1)), but got torch.Size([640, 640]). If you want to instead overwrite randomly initialized weights, please make sure to pass both `low_cpu_mem_usage=False` and `ignore_mismatched_sizes=True`. For more information, see also: https://github.com/huggingface/diffusers/issues/1619#issuecomment-1345604389 as an example.

System Info

• 🤗 Diffusers version: 0.30.0.dev0
• Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
• Running on Google Colab?: No
• Python version: 3.10.12
• PyTorch version (GPU?): 2.3.1+cu121 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Huggingface_hub version: 0.23.2
• Transformers version: 4.40.0
• Accelerate version: 0.31.0
• PEFT version: 0.9.0
• Bitsandbytes version: 0.43.1
• Safetensors version: 0.4.3
• xFormers version: 0.0.27

Who can help?

No response

[Gothos]

Looking at the error closer, it seems it's due to the model type being inferred improperly on inpainting checkpoints. It's being inferred as an SD1.5 model type in the load. I've a PR for this that I'll submit in a little bit.

[DN6]

Hi @Gothos is this a merged inpainting checkpoint? These tend of have keys from both SD-1.5 and SDXL. It's hard to automatically infer them.

Can you try

from diffusers import StableDiffusionXLInpaintPipeline

StableDiffusionXLInpaintPipeline.from_single_file("dreamshaper_inpaint.safetensors", config="diffusers/stable-diffusion-xl-1.0-inpainting-0.1")

[Gothos]

Yes this should also work I guess. I only put in that PR because there was no route to xl_inpaint in from single file, which would have yielded this config.

[Gothos]

https://civitai.com/models/403751/dreamshaper-xl-lightning-inpainting was the checkpoint in question. It'd be nice if from_single_file could handle inpaint checkpoints automatically, is all.