Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to use noisy latent for hires fix #12328

Closed
wants to merge 2 commits into from

Conversation

brkirch
Copy link
Collaborator

@brkirch brkirch commented Aug 5, 2023

Description

This PR adds an option, Hires noisy latent, which sends a noisy latent for the highres pass (without renoising it), which is obtained by producing sigmas for the number of steps for the first pass plus the number of steps for the second pass and then trimming the excess sigmas. Currently this is only implemented for k-diffusion samplers, and it doesn't appear to work correctly for all of them. Also this doesn't really work for increasing resolution, the intended usage for now is sending from the SD XL base model to the refiner without changing resolution.

This should allow for more closely matching the official SD XL workflow when used with #12181.

Screenshots/videos:

Example

Checklist:

Currently only works with k-diffusion samplers.
@brkirch brkirch added the sdxl Related to SDXL label Aug 5, 2023
@brkirch brkirch linked an issue Aug 5, 2023 that may be closed by this pull request
1 task
@AUTOMATIC1111 AUTOMATIC1111 deleted the branch AUTOMATIC1111:hires_checkpoint August 5, 2023 04:47
@AUTOMATIC1111
Copy link
Owner

oh no
I confused this with one of my branches
I didn't meant to delete this

@AUTOMATIC1111
Copy link
Owner

No, I think that was my branch...

Just to clarify, I didn't willingly close this, it was done automatically by github when i deleted the branch I merged.

@AUTOMATIC1111
Copy link
Owner

Looking at the code, I think the correct way to do this is not to add noise after you've finished sampling, but to stop sampling early instead.

My vision for refiner is to make this possible separately from hires fix.

@brkirch
Copy link
Collaborator Author

brkirch commented Aug 5, 2023

@AUTOMATIC1111 Yes, I can confirm that I made the process needlessly complicated. It looks like a working approach is to start sampling for (first pass steps + second pass steps), stop after first pass steps, then run img2img with the refiner for second pass steps without adding noise and also setting denoising strength to (second pass steps) / (first pass steps + second pass steps).

@brkirch brkirch deleted the noisy-latent branch August 16, 2023 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sdxl Related to SDXL
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature Request]: SDXL refiner support
2 participants