Delegated pixel-processing effect

[MV10]

I have an idea that should be relatively easy to implement: use the same cell-based parallel processing technique that I built for motion detection and convolution to invoke a per-pixel delegate to modify each pixel. This would allow the user to write a custom function to do basic operations at the individual pixel level like tinting, grayscale conversion, gamma correction, and so on.

Delegates are thread-safe as long as the operation they perform is thread-safe (which is the case for parallel processing that targets unique "cells" in the image data array). The input would be the pixel data (I still have to ponder RGB24 vs RGB32 vs RGBA) and maybe the X,Y coordinates (unsure if that would be useful). The output would be the modified pixel data to write back to the buffer.

Sort of a poor-man's pixel shader.

This would be especially fun if you can figure out how to allow the effects to run earlier in the pipeline, as we discussed in some of the motion detection threads where I was trying to get it working at the port level.

[MV10]

A related idea: it may also be interesting to look at generalizing the cell-level processing into a delegate approach. If the call overhead isn't too terrible, changes like the convolution code in PR #175 could perhaps just turn into an implementation of that. Although I'll be surprised if the framework doesn't add a lot of overhead. Even so, a cell-level delegate might be interesting, too, even if it's faster to hard-code it internally as the PR does.

[MV10]

Possibly related to #82

[MV10]

I'm playing around with this and a per-pixel delegate is about as fast as the convolution processing -- delegate invocation overhead doesn't seem to be a big problem. We're about to take a vacation but I should have something on this around the middle of next week.

However, I'm realizing the CloneToRawBitmap method I wrote for convolution also results in a BGR buffer, so Bitmap's channel-mixup works in the other direction as well. This means we have two scenarios.

Scenario 1 is when the original context is raw. The data is RGB and we must do the red/blue swap before saving.

Scenario 2 is when the original context is encoded. We need raw data, but Bitmap is returning that as BGR -- and to re-encode, it needs to stay BGR. So instead of swapping twice, we can use an array index offset to point to the red and blue channels and process them in place.

A bit of a headache.

[MV10]

Here's a vignette delegate (linear, so a bit crude), runs in about 500 ms on a 1296 x 972. (v1 mode4) image:

context.Apply(new CustomPixelProcessor((r, g, b, x, y, w, h) =>
{
    var xc = w / 2;
    var yc = h / 2;
    var point = Math.Sqrt(Math.Pow(x - xc, 2) + Math.Pow(y - yc, 2));
    var center = Math.Sqrt(Math.Pow(xc, 2) + Math.Pow(yc, 2));
    var dist = 1.0f - (point / center);
    return ((int)(r * dist), (int)(g * dist), (int)(b * dist));
}));

And a grayscale delegate runs in about 340 ms:

context.Apply(new CustomPixelProcessor((r, g, b, x, y) =>
{
    var i = (int)((r * 0.299f) + (g * 0.587f) + (b * 0.114f));
    return (i, i, i);
}));

Users will have to be careful about thread safety, though. I'm thinking about a couple of ways to allow the user to pass arbitrary data to the delegate but I need to review how to make it thread safe -- I think I can use a generic struct param but I need to double check some assumptions.

From here I think the cell-based delegate will be relatively trivial.

I should really go pack my luggage, though. 😬

Vignette of my office (excuse the mess, we're moving house)...

[MV10]

If you want to have a look at the work-in-progress while I'm out of town, I moved convolutions to a separate folder and put much of the common logic into a base class. The cell-based delegate is currently stubbed until I get back, but:

https://github.com/MV10/MMALSharp/tree/image_fx_delegates/src/MMALSharp.Processing/Processors/Effects

The vignette implementation using a custom metadata struct:

https://github.com/MV10/pi-cam-test/blob/local_mmal_dev_pkg/Program.cs#L854

https://github.com/MV10/pi-cam-test/blob/local_mmal_dev_pkg/VignetteMetadata.cs