Implement Differential Diffusion

[shiimizu]

The authors of the paper introduced a concept called a change map. For brevity, I will refer to it here as a mask.

The node _for_testing/Differential Diffusion is meant to be used alongside the InpaintModelConditioning node or the Set Latent Noise Mask node, and then the sampler does it automatically.

Implementation details

• A full change of the mask is done, essentially $\fbox{mask >= threshold}$ instead of $\fbox{mask > threshold}$. This is important because the contents of the mask should be followed, and for latent inpainting/outpainting (where the mask becomes binary), the initial step should paint something.

General info/tips

• Lighter areas get painted earlier, darker areas later.
  • Try adjusting the mask's blur, opacity, brightness, exposure, etc.
  • To get more of the original image, lower the mask's opacity/brightness or increase its blur.
• The more steps, the finer it's applied (due to finer thresholds).
• Lower your cfg if things look too noisy or fried.

Comparisons

These were made alongside the InpaintModelConditioning node without an inpainting model.

Example workflow

Here's a simple workflow to get started. (Note: It uses some custom nodes).
Feel free to use another blur node.
Also, see the official inpaint examples.

Download this image to drag & drop it in ComfyUI.

Issues

• The callback doesn't seem to get called for UniPC samplers.

---

Resolves #2851, resolves #2671

[comfyanonymous]

This should have fixed the uni_pc issue: 10847df

[gqfx]

could you plz give an example to show how to use this node ? thanks a lot !

[shiimizu]

I updated the original post with an example workflow.

[comfyanonymous]

I thought this was going to break at some point in the future so I simplified the code: 12c1080

Behaviour should be very close to identical as before at least on the normal scheduler, let me know if that's not the case.

[ethanfel]

I thought this was going to break at some point in the future so I simplified the code: 12c1080

Behaviour should be very close to identical as before at least on the normal scheduler, let me know if that's not the case.

Hi, I just moved back from the shiimizu fork, removed the nodes and readded them and it return an error.

[comfyanonymous]

can I see your full stacktrace?

[shiimizu]

It's when batch_size > 1. https://github.com/comfyanonymous/ComfyUI/blob/12c1080ebc9095b3878d7b3aa994a6b75b308e0a/comfy_extras/nodes_differential_diffusion.py#L30
In that line, I think sigma should be sigma[0]

[comfyanonymous]

Yes that should be fixed now.

[kadirnar]

Yes that should be fixed now.

'UNetModel' object has no attribute 'default_image_only_indicator'

File "D:\ComfyUI\ComfyUI\custom_nodes\FreeU_Advanced\nodes.py", line 159, in __temp__forward
image_only_indicator = kwargs.get("image_only_indicator", self.default_image_only_indicator)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1688, in __getattr__
raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")

[ltdrdata]

Yes that should be fixed now.

'UNetModel' object has no attribute 'default_image_only_indicator'

File "D:\ComfyUI\ComfyUI\custom_nodes\FreeU_Advanced\nodes.py", line 159, in __temp__forward
image_only_indicator = kwargs.get("image_only_indicator", self.default_image_only_indicator)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1688, in __getattr__
raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")

disable FreeU_Advanced

[DENG-XC]

Hi, thanks you for your guys hard work. This is a great new inpainting node, but I have problem with this node currently. differential diffusion is not working with controlnet and IPAdapter？ The output image get burn when I connect controlnet and differential diffusion together. (IPAdapter have the same problem)