How to use CircularBufferCaptureHandler combined with another (continious capture) handler

[maartendweerdt]

Hi,

First of all I would like to thank you (@techyian) for creating this library! It's great!

Secondly I'm having some issues connecting a CircularBufferCaptureHandler and my own handler together.

Grand picture:
What I'm trying to accomplish is a program that when it detects motion it will send the frame to a computer vision algorithm to identify the motion. If the computer vision algorithm returns a specific value I open up the garden hoze and a second video (capturing the whole scaring away the pigeon part) should be send to the cloud.

Using some of the examples in the docmentation and some answers to the issues here, I was able to create this, which works perfectly for the motiondetection with Emgu.CV:

(source: https://github.com/techyian/MMALSharp/issues/117)

private static async Task TakeVideoAsync()
        {
            MMALCamera camera = MMALCamera.Instance;
            MMALCameraConfig.VideoResolution = new Resolution(2240, 1680);
            MMALCameraConfig.VideoEncoding = MMALEncoding.BGR24;

            using (var vidCaptureHandler = new EmguInMemoryCaptureHandler())
            using (var splitter = new MMALSplitterComponent())
            using (var imgEncoder = new MMALImageEncoder(continuousCapture: true))
            using (var renderer = new MMALNullSinkComponent())
            {
                camera.ConfigureCameraSettings();

                vidCaptureHandler.MyEmguEvent += RunPigeonShooter;

                var portConfig = new MMALPortConfig(MMALEncoding.BMP, MMALEncoding.BGR24, 90);

                // Create our component pipeline.         
                imgEncoder.ConfigureOutputPort(portConfig, vidCaptureHandler);

                camera.Camera.VideoPort.ConnectTo(splitter);
                splitter.Outputs[0].ConnectTo(imgEncoder);
                camera.Camera.PreviewPort.ConnectTo(renderer);

                // Camera warm up time
                await Task.Delay(2000);

                CancellationTokenSource cts = new CancellationTokenSource();

                // Process images       
                await camera.ProcessAsync(camera.Camera.VideoPort, cts.Token);
            }

            camera.Cleanup();
        }

        protected static void  RunPigeonShooter(object sender, ImageFrameEventArgs args)
        {
            //Check for motion using Emgu.CV
            byte[] frame = MotionDetector.ProcessImage(args.ImageData);

            if (frame != null)
            {
                   //Just a http call to a REST API that will return a prediction score
                   var isPigeon = PigeonModel.Score(frame);
                    if (isPigeon){
                            //Open up garden hoze
                            controller.Write(GpioPin, PinValue.High);                            
                            Thread.Sleep(2000);
                            controller.Write(GpioPin, PinValue.Low);
                            Thread.Sleep(2000);
                            
                            //Here I would like to have the ability to save the previous 10s of video footage

                        }
                    }
                }                
            }
        }

I've been trying to connect an additional (private static) CircularBufferCaptureHandler to continiously capture the videostream. However I can't get it to work. Receiving a lot of mmal_port errors.

What I tried was something like this:

        private static CircularBufferCaptureHandler circularVidHandler = new CircularBufferCaptureHandler(6291456, "/home/pi/videos/", "mjpeg");
        static void Main(string[] args)
        {
            TakeVideoAsync();
            circularVidHandler.StartRecording();
            Console.ReadLine();
        }

        private static async Task TakeVideoAsync()
        {
            MMALCamera camera = MMALCamera.Instance;
            MMALCameraConfig.VideoResolution = new Resolution(2240, 1680);
            MMALCameraConfig.VideoEncoding = MMALEncoding.BGR24;

            using (var vidCaptureHandler = new FrameInMemoryCaptureHandler())
            using (var splitter = new MMALSplitterComponent())
            using (var vidEncoder = new MMALVideoEncoder())
            using (var imgEncoder = new MMALImageEncoder(continuousCapture: true))
            using (var renderer = new MMALVideoRenderer())
            {
                camera.ConfigureCameraSettings();
                vidCaptureHandler.MyFrameEvent += RunPigeonShooter;

                var portConfig = new MMALPortConfig(MMALEncoding.BMP, MMALEncoding.BGR24, 90);
                var vidPortConfig = new MMALPortConfig(MMALEncoding.MJPEG, MMALEncoding.I420, 0, MMALVideoEncoder.MaxBitrateMJPEG, null);
                // Create our component pipeline.         
                imgEncoder.ConfigureOutputPort(portConfig, vidCaptureHandler);
                vidEncoder.ConfigureOutputPort(vidPortConfig, circularVidHandler);

                camera.Camera.VideoPort.ConnectTo(splitter);
                splitter.Outputs[0].ConnectTo(imgEncoder);
                splitter.Outputs[1].ConnectTo(vidEncoder);

                camera.Camera.PreviewPort.ConnectTo(renderer);

                // Camera warm up time
                await Task.Delay(2000);

                CancellationTokenSource cts = new CancellationTokenSource();

                // Process images       
                await camera.ProcessAsync(camera.Camera.VideoPort, cts.Token);
            }

            camera.Cleanup();
        }

        protected static void  RunPigeonShooter(object sender, ImageFrameEventArgs args)
        {
            //Check for motion using Emgu.CV
            byte[] frame = MotionDetector.ProcessImage(args.ImageData);

             if (frame != null)
            {
                   //Just a http call to a REST API that will return a prediction score
                   var isPigeon = PigeonModel.Score(frame);
                    if (isPigeon){
                            //Open up garden hoze
                            controller.Write(GpioPin, PinValue.High);                            
                            Thread.Sleep(2000);
                            controller.Write(GpioPin, PinValue.Low);
                            Thread.Sleep(2000);
                            circularVidHandler.StopRecording();
                            circularVidHandler.StartRecording();
                        }
                    }
                }                
            }
        }

Is it possible to get a small example of how I could use the CircularBufferCaptureHandler in addition to another handler?

Thanks in advance!

[techyian]

Hey, thanks for the feedback, it's really appreciated.

@MV10 has been making some great progress improving the motion detection feature built into the library. We now have low latency motion detection and are able to record video / take still photos when motion is detected. It's still a work in progress and is only available in the current dev branch but I'd recommend giving that a try first. If you want an example which shows how to take still images and record video, you can check out Jon's example in #169. For reference there's also #175 and the wiki example on motion detection here.

Hopefully the example in #169 will get you close enough to where you're looking to be, but please let me know if you're struggling with anything.

[maartendweerdt]

Hey Ian,

Yes this example is kinda what I was looking for, thanks for the reference - should have found that one myself.

With this example I got it to work with version 0.6.0. I'll post my solution below if anyone is looking to do something similar.
Many thanks!

//private static CircularBuffer so you can use this handler outside the camera startup code
private static CircularBufferCaptureHandler vidCaptureHandler = new CircularBufferCaptureHandler(1000000, "/home/pi/videos/", "h264");
private static async Task TakeVideoAsync()
        {
            MMALCamera cam = MMALCamera.Instance;
            MMALCameraConfig.InlineHeaders = true;
            MMALCameraConfig.VideoResolution = Resolution.As1080p;

            cam.ConfigureCameraSettings();

            //Using own CaptureHandler (as described in issue #117 )
            using (var imgCaptureHandler = new FrameInMemoryCaptureHandler())

            // this will feed the image capture handler above
            using (var imgEncoder = new MMALImageEncoder(continuousCapture: true))

            // typical for a splitter example            
            using (var splitter = new MMALSplitterComponent())
            using (var vidEncoder = new MMALVideoEncoder())
            using (var renderer = new MMALVideoRenderer())
            {
                imgCaptureHandler.MyFrameEvent += ProcessFrame;
                // typical for a splitter example
                splitter.ConfigureInputPort(new MMALPortConfig(MMALEncoding.OPAQUE, MMALEncoding.I420), cam.Camera.VideoPort, null);
                vidEncoder.ConfigureOutputPort(new MMALPortConfig(MMALEncoding.H264, MMALEncoding.I420, 0, MMALVideoEncoder.MaxBitrateLevel4, null), vidCaptureHandler);

                // send image encoder output to the image capture handler
                imgEncoder.ConfigureOutputPort(new MMALPortConfig(MMALEncoding.BMP, MMALEncoding.BGR24, quality: 90), imgCaptureHandler);

                // typical for a splitter example
                cam.Camera.VideoPort.ConnectTo(splitter);
                cam.Camera.PreviewPort.ConnectTo(renderer);
                splitter.Outputs[0].ConnectTo(vidEncoder);
                splitter.Outputs[1].ConnectTo(imgEncoder);

                // camera warmup
                await Task.Delay(2000);

                vidCaptureHandler.StartRecording();
                // processing for 30 seconds
                var cts = new CancellationTokenSource();
                await cam.ProcessAsync(cam.Camera.VideoPort, cts.Token);

            }

            cam.Cleanup();
        }

        protected static void ProcessFrame(object sender, ImageFrameEventArgs args)
        {
            //process each frame (for example to give it as input to EmguCV) 
            //-- omitted details which are specific to me --
            
           //Record clip - make sure that you 'pause' in code above if you need to capture a reaction to your code (Thread.sleep does the trick)
            vidCaptureHandler.StopRecording();
            vidCaptureHandler.Split();
            vidCaptureHandler.NewFile();
            vidCaptureHandler.StartRecording();
        }

[maartendweerdt]

Is it possible that the V0.6 version of CircularBufferCaptureHandler generates an OutOfRangeException/OutOfMemoryExceptions due to the byte array not being overwritten but constantly extended until you hit the max size?

I see this has been resolved with the changes in the dev branch, any idea when v0.7 will be released?

[techyian]

Not that I recall, do you have a commit ID that you think it was fixed in? I don't have an ETA for 0.7, I'm very busy with other work at the moment and there are some rather large pieces of work I was hoping to get into 0.7 but these may have to be pushed back. I'd like to get the motion detection improvements stable before releasing anything. Pre-release builds can be found on MyGet.

[maartendweerdt]

Not that I recall, do you have a commit ID that you think it was fixed in? I don't have an ETA for 0.7, I'm very busy with other work at the moment and there are some rather large pieces of work I was hoping to get into 0.7 but these may have to be pushed back. I'd like to get the motion detection improvements stable before releasing anything. Pre-release builds can be found on MyGet.

Seems I was looking on Branch V0.6 instead of Master which contains the current nugget. so ignore my last comment.

[techyian]

Yes, I need to remove those branches so thanks for reminding me. Master is latest stable in NuGet or you can also use the tags.

[techyian]

Branches removed. Closing.