Pixel remapping/Swizzling

[onepiecefreak3]

This feature is implemented in many graphics hardwares at least of most gaming consoles. Adding this feature would enable ImageSharp to natively allow for these specifications of graphics hardware to be supported out of the box.

As per OpenCV naming and implementation this feature is most likely known as "Remapping" or "Point remapping". So as the naming goes, it should incorporate either one.

Point remappings are, as the name suggests, a process to rearrange the pixels from a source to a destination point, where each source point is modified as per following method:
dst(x,y) = src(fx(x,y), fy(x,y))

As for the implementation following this function, it should allow executing an action for both the x- and y-coordinate, and return its modified value respectively. Or an even simpler approach as seen in the Kanvas library of the Kuriimu2 project, where it just takes in a Point and returns a new one with the modified coordinates.

Sources:
OpenCV documentation on remapping: https://docs.opencv.org/2.4/modules/imgproc/doc/geometric_transformations.html?highlight=remap
Kuriimu2's Kanvas Swizzle interface:https://github.com/FanTranslatorsInternational/Kuriimu2/blob/dev/src/Kontract/Kanvas/IImageSwizzle.cs

[antonfirsov]

As far as I remember our discussion, remapping should actually alter the image. This could be implemented easily in a single ImageProcessor that consumes an IImageSwizzle-like interface as a parameter.

For a green hand,it's difficult to operate,as i don't know the mean of this.

[Sergio0694]

This also seems a pretty good candidate to use the value delegate trick (for those unfamiliar, I mean using a struct type implementing an interface and making the method generic with the T : struct, ISomeInterface constraint, instead of taking an actual delegate like a Func<...>), since if I'm reading that linked OpenCV correctly, you'd basically have to invoke two custom functions to map the x, y coordinates of every single pixel in the original image 🤔
If there were two non-inlineable callvirt for each pixel, that'd be quite the performance hit.

so,i can use it to do what

[Mahiar7]

It was very helpful

[louis126]

Good ideas.

[onepiecefreak3]

@Sergio0694 Of course having 2 delegates calling for each coordinate x and y could hit performance in the situation you described. That's why I also included a sample from Kanvas, where a Point struct is just taken in, modified, and then returned. That might at least reduce the delegate count to one. But however you want to implement it, is of course the decision of this team here. I just gave my input on already existing examples for this kind of operation.

[Evangelink]

Not sure to fully understand what's supposed to be done for this ticket. My understanding is that we want to:

1. Create an IImageSwizzle-like interface

2. Create a new generic Processor class

3. Add a ctor with this Func signature

4. Do the processing in the OnFrameApply

Am I right?

[antonfirsov]

@Evangelink yes, the steps are mostly correct, but a few key things missing, for example the extension method. I would do the following:

Essential:

1. Define IImageSwizzle interface
2. Define a non-generic SwizzleProcessor : IImageProcessor class that will act as a factory for the generic processor. (See other processor implementations.)
3. Define a generic SwizzleProcessor<TPixel> class and implement OnFrameApply
4. Define an extension method over the non-generic IImageProcessingContext that takes an IImageSwizzle. This can be used with Mutate and Clone. (See other extension methods delegating to processors.)

API sugar:

5. Define an internal DefaultSwizzle : IImageSwizzle type that takes a Func<Point, Point> in constructor
6. Define a public extension method over IImageProcessingContext that takes Func<Point, Point> and delagates to DefaultSwizzle. Note: this stuff will be relatively slow.

Regression tests:

7. Define regression tests as in other processors, but instead of validating with reference outputs, validate the result by manually "Unswizzling" the image in the test code, and comparing the result to the original.

Performance sugar from #1115 (comment):

8. Alter the the non-generic SwizzleProcessor to SwizzleProcessor<TSwizzle> where TSwizzle : struct, IImageSwizzle
9. Similarly, change the generic SwizzleProcessor<TPixel> to SwizzleProcessor<TImageSwizzle, TPixel>. All calls to the TImageSwizzle member will be inlined then, which improves speed.
10. Alter the extension method that takes IImageSwizzle to:

public static Swizzle<TImageSwizzle>(this IImageProcessingContext context, TImageSwizzle swizzle)
    where TImageSwizzle : struct, IImageSwizzle { ... }

OFC you may decide to add 8-10 to your implementation immediately.

[Sergio0694]

FYI @Evangelink in case you were not familiar with these "value delegates" (constrained struct + interface), to add to @antonfirsov's point with respect to 5. and 8., the idea is that once you add the generic constraint, you'd no longer use a Func<Point, Point> at all, but instead inject that functionality through an interface method on that IImageSwizzle interface, so that the JIT compiler can see the actual type in the generic instantiation of that processor type and devirtualize/inline it 😊

You do not want to define a struct with that interface that just wraps some delegate, otherwise that'd defeat the whole point of this optimization (as those calls would remain virtual, just with even more steps due to the extra generic argument).

As in (just an example to illustrate the general idea, ignore the actual type names):

// Before
class DefaultSwizzle
{
    public DefaultSwizzle(Func<Point, Point> swizzler) { }
}

// After
interface ISwizzler
{
    Point Swizzle(Point point);
}

class DefaultSwizzle<TSwizzler>
    where TSwizzler : struct, ISwizzler
{
    public DefaultSwizzle(TSwizzler swizzler) { }
}

And following this, you'd have to make the following changes when actually using these types:

// Before
var defaultSwizzle = new DefaultSwizzle(xy => /* Some logic... */ xy);

// After
struct MySwizzler : ISwizzler
{
    Point Swizzle(Point point) => /* Some logic... */ point;
}

var defaultSwizzle = new DefaultSwizzle<MySwizzler>(default);

If you need more examples of this pattern, you can browse the various row processors we use in ImageSharp here and that are leveraged by all the existing image processors, they do the same trick with a "value delegate" implemented as a struct + interface.

Hope this helps! 😄