CLIP skip experiment on style coadapter

[hafriedlander]

I noticed sometimes when using style images that semantic elements from the style image would affect the result - for instance, a photo with many people would result in people in the output, even if the sketch image and prompt are for something else (i.e. a motorcycle).

I wondered if using earlier hidden layers from the clip image encoding would help.

The answer is "sort of". It's a sometimes useful technique, but also has some drawbacks.

Here's a couple of examples:

Starry night style, final layer results in many stars in the sky, penultimate layer reduces them while still giving a painterly style (but the colors shift too, undesirably)

Style source:

Final hidden layer:

Penultimate layer:

A random image from unsplash, when using the "Deliberate" model you get a lot of the houses from the style image appearing in the result. Penultimate still gives a similar style without repeating those elements. This is probably my best success so far with this technique:

Style source:

Final hidden layer:

Penultimate layer:

[ToTheBeginning]

Interesting finding, you can also adjust the Condition weight of the style to see if it helps.

I tested your second case in CoAdapter, following are the result when Condition weight = 0.9 and prompt = ""

and when Condition weight = 1.0 and prompt = ""

and when Condition weight = 0.85 and prompt = "motorcycle"