Continuous stream capture

[feonx]

Hi, I need some advice on how to proceed.

What I'm trying to do is to continuously capture a stream of the camera input so I can extract the colors used. In order to do this I tried the basic example in the Wiki "Image capture" and instead of the ImageStreamCaptureHandler I used the InMemoryStreamCaptureHandler.

This works but seems to be really slow (2 seconds) for what I want to achieve.

After some searching the Wiki I found the "Advanced-Examples" and I try to use the "Rapid image capture" with the InMemoryCaptureHandler. This however I cannot get to work. I fired the call "cam.ProcessAsync(cam.Camera.VideoPort, cts.Token);" without the await because I do not want to wait, but reading and processing the memory stream while this runs seem not to work.

Perhaps the memory stream is in use? or the ProcessAsync is not finished with the capture and I'm half-way reading it? I'm not sure.

Can someone advice me on what the best approach is to achieve my goal? Or what is the best way so I can quickly capture into the memory (in low quality) and convert the data to a byte array?

Thanks in advance,
Mike

[techyian]

Hi,

For this you will want to create your own Capture handler (or extend from InMemoryCaptureHandler and invoke a callback when you receive a full image frame, either using an event or Action<byte[]>. If you're not calling await, you're basically doing a fire and forget, are you sure that's what you want? You don't need to provide a CancellationToken if you don't want to and the capturing will continue indefinitely. What type of application are you calling this code from? Console, GUI or Web app?

I have provided an example in this ticket on how to use an event callback to retrieve the image data as a byte array when a full frame has been processed by the capture handler - you can use the same logic and specify that capture handler to the example here.

If you're still having issues, please provide a code snippet so I can see what you're doing.

[feonx]

Hi Techyian,

Thanks for your help! It got me on the good track.

However, I tried to use your example but noticed it is a bit outdated? Looking at the InMemoryCaptureHandler I noticed the 'eos' boolean is removed?
https://github.com/techyian/MMALSharp/blob/v0.5.1/src/MMALSharp.Processing/Handlers/InMemoryCaptureHandler.cs

Here is my custom capture handler:
`public class InMemoryFrameCaptureHandler : InMemoryCaptureHandler
{
public event EventHandler OnFrameHandled;

    public override void Process(byte[] data)
    {
        base.Process(data);
        OnFrameHandled?.Invoke(this, EventArgs.Empty);            
    }
}`

Here is the caller/usage:
`public class HdmiCapturer : IHdmiCapturer
{
private readonly MMALCamera _camera;
private readonly CancellationTokenSource _tokenSource;

    private ILogger _logger;
    public HdmiCapturer(ILogger logger)
    {
        _camera = MMALCamera.Instance;
        _tokenSource = new CancellationTokenSource();     
        // _tokenSource.CancelAfter(1500);
        _logger = logger;
    }

    public event EventHandler<Bitmap> OnCaptured;
    public async void StartCapture()
    {
        var memoryCaptureHandler = new InMemoryFrameCaptureHandler();
        
        _logger.LogInfo("Starting camera capture...");
         await CaptureFast(memoryCaptureHandler);               

        memoryCaptureHandler.OnFrameHandled += async (sender, args) =>
        {
            
             _logger.LogInfo("Saving to memory..");
             await using var memoryStream = new MemoryStream(memoryCaptureHandler.WorkingData.ToArray());
            
             _logger.LogInfo("Saving to bitmap");
             var bitmap = new Bitmap(memoryStream);

             OnCaptured?.Invoke(this, bitmap);
        };
    }

    private async Task CaptureFast(ICaptureHandler handler)
    {
       using var splitter = new MMALSplitterComponent();
       using var imageEncoder = new MMALImageEncoder(handler, continuousCapture: true); 
       using var nullSink = new MMALNullSinkComponent();

       _camera.ConfigureCameraSettings();
       
       var portConfiguration = new MMALPortConfig(MMALEncoding.JPEG, MMALEncoding.I420, 10);

       imageEncoder.ConfigureOutputPort(portConfiguration);

       _camera.Camera.VideoPort.ConnectTo(splitter);
       splitter.Outputs[0].ConnectTo(imageEncoder);
       _camera.Camera.PreviewPort.ConnectTo(nullSink);

       await Task.Delay(2000);

        await _camera.ProcessAsync(_camera.Camera.VideoPort, _tokenSource.Token);
    }
}`

I tried to use the event handling so I can use that as a trigger.

Currently this code works but will eventually result in an out-of-memory exception. This is because I think the byte array buffer is never cleared. How can I check whether the frame is done so I can clear the byte buffer and trigger an event?

BTW: It is a backend service (WebAPI)
Thanks

[techyian]

I'm really trying to push users towards the 0.6 version of the code, which can either be cloned from here, or if you'd prefer a NuGet package, there are pre-release builds over at MyGet. Using the latest code will allow you to use the example I provided and will give you access to the eos flag. It's also had a number of improvements added, especially around the async code.

I am currently working on #47 and then will be in a position to push out v0.6 officially. I hope this won't be too long off now, but the rawcam work is quite complex and I want to make sure it's working properly before I push anything out.

[feonx]

Thank you very much Techyian! It works like a charm now. Keep up the good work!

[techyian]

That's great :) thanks very much.