Shader for PPSSPP's Hybrid Method

[Papermanzero]

The Hybrid method from PPSSPP combines the sharp color transitions from xbr with smooth gradiants.

The algorithm is as follows:

1. determine a feature mask C based on a sobel-ish filter + splatting, and upscale that mask bilinearly
2. generate 2 scaled images: A - using Bilinear filtering, B - using xBRZ
3. output = A_C + B_(1-C)

Step 1) and 3) needs a shader file to create the Hybrid look.
For 2) xBR/ Super xBR / or any other filter can be used.

PPSSPP Source Header:
https://github.com/hrydgard/ppsspp/blob/c2f4fad56aada331bbd8dd7db99037285bab853f/GPU/Common/TextureScalerCommon.h

PPSSPP Source CPP:
https://github.com/hrydgard/ppsspp/blob/c2f4fad56aada331bbd8dd7db99037285bab853f/GPU/Common/TextureScalerCommon.cpp

[hizzlekizzle]

can you get some screenshots for reference?

[Papermanzero]

There is also a blog entry about the technique.
http://blog.metaclassofnil.com/?tag=textures
http://blog.metaclassofnil.com/?p=306

Some Screeshots:
http://i.imgur.com/24xh3tG.png
http://i.imgur.com/gPAwj5m.png
http://i.imgur.com/2ve8nXM.png
http://i.imgur.com/c5VXbI8.png

http://abload.de/img/gedosatotexzmo3e.jpg

The posterisation effect of xbr is very reduced.
Monster Hunter Portable is one of the best examples. However I don't find screenshots for that.
So I have to take some of my own.

[andres-asm]

Hmm this isn't really possible in the shader pipeline.
Our shaders apply post-processing to the finished output.

What you want is specific to the emulator and what it actually does is apply post-processing to the textures before applying them to the final output.

[Papermanzero]

I don't want texture filtering within the shader pipeline.
I would like to have the 'hybrid' method to filter the final image output as a shader
(post-processing effect). Means to filter the final complete picture with the hybrid shader.
The post processing effect shall follow the algorithm which I posted in the initial message.

As an example: In case of a 2D game, PPSSPP and the Retroarch Hybrid Filter should lead to the same result. In case of a 3D game, PPSSPP will only filter the textures but the the Retroarch Hybrid Filter will filter the complete image.

Unfortunately I only have examples from 3D games.
I guess this was a little bit confusing.

[Papermanzero]

Now I understand your post. :)

The limitation in retroarch is the iterative approach. So you can only apply a certain processing on the final picture, which will be the input for the next stage.
Means:
Image 1 -> Processing (e.g. Bilinear) -> Image 2 - > Processing (e.g. SmartBlur) -> Image 3

However this filter needs a complex algorithm.
Means:
Image 1 -> Processing (Bilinear) -> Storage of Result 1
Image 1 -> Processing (xBR) -> Storage of Result 2
Image 1 -> Processing (Sobel) -> Storage of Result 3
Image 2 = Result 1 x Result 3 + Result 2 (1 - Result 3)

Maybe a recursive approach could be helpful in general for the future.
This would allow much complex filters.

[hizzlekizzle]

RetroArch is capable of referencing previous passes. We also already have a variety of upscalers and even some edge-detection shaders. It should just be a matter of running each one in a pass and then making a pass that says "if it was an edge, show the xbr result; if not, show bilinear (or whatever)". I'll try to take a stab at it soon.

[hizzlekizzle]

Alright, I got something worked up in slang but I can port it to Cg pretty easily if it does what you're looking for. I based it on sabr instead of xbr because I already had a single-pass slang version of sabr handy but I can base on whatever in Cg:
sabr alone: http://i.imgur.com/wfXoV4X.png
sabr-hybrid-deposterize: http://i.imgur.com/IXVOz5C.png

[Papermanzero]

Wow this looks awesome. Especially the skin texture looks great.
I hope slang will replace cg soon. But for now cg is still the reference.
So it would be great if you could port it to cg.

Thank you so much.