Skip to content
This repository has been archived by the owner on Feb 22, 2024. It is now read-only.

Advanced Examples

Jump to bottom
Ian Auty edited this page Jan 2, 2020 · 40 revisions

Revisions

  • v0.6 Alpha (Current)
  • v0.5

Contents

  1. Rapid image capture
  2. Raw video from resizer
  3. Raw video from splitter
  4. Raw video from resizer with splitter component
  5. Encode / Decode from Stream - Image
    1. Encode
    2. Decode
  6. Encode / Decode from Stream - Video
    1. Encode
    2. Decode
    3. Decode -> Encode
    4. Decode -> Splitter -> Encode
    5. Decode -> Splitter -> Resizer -> Encode
  7. Static render overlay
  8. FFmpeg - RTMP streaming
  9. FFmpeg - Raw video convert
  10. FFmpeg - Images to video

Notes

FFmpeg

For FFmpeg functionality, you will need to install the latest version of FFmpeg from source - do not install from the Raspbian repositories as they don't have H.264 support.

A guide to installing FFmpeg from source including the H.264 codec can be found here

Rapid image capture

By utilising the camera's video port, we are able to retrieve image frames at a much higher speed than using the conventional still port. Images captured via the video port will be of a lesser quality and do not support EXIF.

public async Task TakePictureFromVideoPort()
{                        
    MMALCamera cam = MMALCamera.Instance;

    using (var imgCaptureHandler = new ImageStreamCaptureHandler("/home/pi/images/", "jpg"))
    using (var splitter = new MMALSplitterComponent())
    using (var imgEncoder = new MMALImageEncoder(continuousCapture: true))
    using (var nullSink = new MMALNullSinkComponent())
    {
        cam.ConfigureCameraSettings();
        
        var portConfig = new MMALPortConfig(MMALEncoding.JPEG, MMALEncoding.I420, 90);

        // Create our component pipeline.         
        imgEncoder.ConfigureOutputPort(portConfig, imgCaptureHandler);
                
        cam.Camera.VideoPort.ConnectTo(splitter);
        splitter.Outputs[0].ConnectTo(imgEncoder);                    
        cam.Camera.PreviewPort.ConnectTo(nullSink);
        
        // Camera warm up time
        await Task.Delay(2000);
                
        CancellationTokenSource cts = new CancellationTokenSource(TimeSpan.FromSeconds(15));
        
        // Process images for 15 seconds.        
        await cam.ProcessAsync(cam.Camera.VideoPort, cts.Token);
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    cam.Cleanup();
}

Raw video capture from resizer component

Available in v0.5.1

The resizer component can adjust the resolution coming from the camera's video port, allowing you to record raw YUV420 frames. By passing VideoPort to the generic constraint on ConfigureOutputPort, we tell MMALSharp to use VideoPort behaviour on the resizer's output. By default, the resizer uses StillPort behaviour and would not work in this scenario.

public async Task RecordVideoDirectlyFromResizer()
{                        
    MMALCamera cam = MMALCamera.Instance;

    using (var vidCaptureHandler = new VideoStreamCaptureHandler("/home/pi/videos/tests", "raw"))
    using (var resizer = new MMALResizerComponent())
    using (var preview = new MMALVideoRenderer())
    {
        cam.ConfigureCameraSettings();
    
        // Use the resizer to resize 1080p to 640x480.
        var portConfig = new MMALPortConfig(MMALEncoding.I420, MMALEncoding.I420, 640, 480, 0, 0, 0, false, null);

        resizer.ConfigureOutputPort<VideoPort>(0, portConfig, vidCaptureHandler);

        // Create our component pipeline.         
        cam.Camera.VideoPort
            .ConnectTo(resizer);
        cam.Camera.PreviewPort
            .ConnectTo(preview);

        // Camera warm up time
        await Task.Delay(2000);

        CancellationTokenSource cts = new CancellationTokenSource(TimeSpan.FromSeconds(15));

        // Record video for 20 seconds
        await cam.ProcessAsync(cam.Camera.VideoPort, cts.Token);
    }
    
    // Only call when you no longer require the camera, i.e. on app shutdown.
    cam.Cleanup();
}

Raw video capture from splitter component

Available in v0.5.1

The splitter component can also be used to record raw video frames. As seen with the resizer example above, we can pass VideoPort as a generic constraint to the ConfigureOutputPort method, instructing MMALSharp to use VideoPort behaviour for the splitter's output port. If no type is passed in, the splitter will simply act as a pass-through component.

public async Task RecordVideoDirectlyFromSplitter()
{                        
    MMALCamera cam = MMALCamera.Instance;

    using (var vidCaptureHandler = new VideoStreamCaptureHandler("/home/pi/videos/tests", "raw"))
    using (var vidCaptureHandler2 = new VideoStreamCaptureHandler("/home/pi/videos/tests", "raw"))
    using (var vidCaptureHandler3 = new VideoStreamCaptureHandler("/home/pi/videos/tests", "raw"))
    using (var vidCaptureHandler4 = new VideoStreamCaptureHandler("/home/pi/videos/tests", "raw"))
    using (var preview = new MMALVideoRenderer())
    using (var splitter = new MMALSplitterComponent())
    {
        cam.ConfigureCameraSettings();
    
        var splitterPortConfig = new MMALPortConfig(MMALEncoding.I420, MMALEncoding.I420, 0, 0, null);

        // Create our component pipeline.         
        splitter.ConfigureInputPort(new MMALPortConfig(MMALEncoding.OPAQUE, MMALEncoding.I420), cam.Camera.VideoPort, null);
        splitter.ConfigureOutputPort<VideoPort>(0, splitterPortConfig, vidCaptureHandler);
        splitter.ConfigureOutputPort<VideoPort>(1, splitterPortConfig, vidCaptureHandler2);
        splitter.ConfigureOutputPort<VideoPort>(2, splitterPortConfig, vidCaptureHandler3);
        splitter.ConfigureOutputPort<VideoPort>(3, splitterPortConfig, vidCaptureHandler4);
        
        cam.Camera.VideoPort.ConnectTo(splitter);
        cam.Camera.PreviewPort.ConnectTo(preview);

        // Camera warm up time
        await Task.Delay(2000);

        CancellationTokenSource cts = new CancellationTokenSource(TimeSpan.FromSeconds(15));

        // Record video for 20 seconds
        await cam.ProcessAsync(cam.Camera.VideoPort, cts.Token);
    }
    
    // Only call when you no longer require the camera, i.e. on app shutdown.
    cam.Cleanup();
}

Raw video from resizer with splitter component

Available in v0.5.1

You can combine both previous examples into one by using the resizer component with the splitter. By combining the components, you can potentially resize up to 4 separate raw video streams which adds a lot of flexibility to your application.

public async Task RecordVideoDirectlyFromResizerWithSplitterComponent()
{                        
    MMALCamera cam = MMALCamera.Instance;

    using (var vidCaptureHandler = new VideoStreamCaptureHandler("/home/pi/videos/tests", "raw"))
    using (var vidCaptureHandler2 = new VideoStreamCaptureHandler("/home/pi/videos/tests", "raw"))
    using (var vidCaptureHandler3 = new VideoStreamCaptureHandler("/home/pi/videos/tests", "raw"))
    using (var vidCaptureHandler4 = new VideoStreamCaptureHandler("/home/pi/videos/tests", "raw"))
    using (var preview = new MMALVideoRenderer())
    using (var splitter = new MMALSplitterComponent())
    using (var resizer = new MMALResizerComponent())
    using (var resizer2 = new MMALResizerComponent())
    using (var resizer3 = new MMALResizerComponent())
    using (var resizer4 = new MMALResizerComponent())
    {
        cam.ConfigureCameraSettings();
    
        var splitterPortConfig = new MMALPortConfig(MMALEncoding.OPAQUE, MMALEncoding.I420, 0, 0, null);

        var portConfig = new MMALPortConfig(MMALEncoding.I420, MMALEncoding.I420, 1024, 768, 0, 0, 0, false, DateTime.Now.AddSeconds(20));
        var portConfig2 = new MMALPortConfig(MMALEncoding.I420, MMALEncoding.I420, 800, 600, 0, 0, 0, false, DateTime.Now.AddSeconds(20));
        var portConfig3 = new MMALPortConfig(MMALEncoding.I420, MMALEncoding.I420, 640, 480, 0, 0, 0, false, DateTime.Now.AddSeconds(15));
        var portConfig4 = new MMALPortConfig(MMALEncoding.I420, MMALEncoding.I420, 320, 240, 0, 0, 0, false, DateTime.Now.AddSeconds(20));

        // Create our component pipeline.         
        splitter.ConfigureInputPort(new MMALPortConfig(MMALEncoding.OPAQUE, MMALEncoding.I420), cam.Camera.VideoPort, null);
        splitter.ConfigureOutputPort(0, splitterPortConfig, null);
        splitter.ConfigureOutputPort(1, splitterPortConfig, null);
        splitter.ConfigureOutputPort(2, splitterPortConfig, null);
        splitter.ConfigureOutputPort(3, splitterPortConfig, null);
        
        resizer.ConfigureOutputPort<VideoPort>(0, portConfig, vidCaptureHandler);
        resizer2.ConfigureOutputPort<VideoPort>(0, portConfig2, vidCaptureHandler2);
        resizer3.ConfigureOutputPort<VideoPort>(0, portConfig3, vidCaptureHandler3);
        resizer4.ConfigureOutputPort<VideoPort>(0, portConfig4, vidCaptureHandler4);

        // Create our component pipeline.         
        cam.Camera.VideoPort.ConnectTo(splitter);

        splitter.Outputs[0].ConnectTo(resizer);
        splitter.Outputs[1].ConnectTo(resizer2);
        splitter.Outputs[2].ConnectTo(resizer3);
        splitter.Outputs[3].ConnectTo(resizer4);

        cam.Camera.PreviewPort.ConnectTo(preview);

        // Camera warm up time
        await Task.Delay(2000);

        await cam.ProcessAsync(cam.Camera.VideoPort);
    }
    
    // Only call when you no longer require the camera, i.e. on app shutdown.
    cam.Cleanup();
}

Encode / Decode from Stream - Image

MMALSharp provides the ability to encode/decode images fed from Streams. It supports GIF, BMP, JPEG and PNG file formats, and decoding must be carried out to the following:

  • JPEG -> YUV420/422 (I420/422)
  • GIF -> RGB565 (RGB16)
  • BMP/PNG -> RGBA

Note: Please notice the <FileEncodeOutputPort> generic constraint when calling .ConfigureOutputPort, this is an important addition as this port has the ability to handle MMAL_EVENT_FORMAT_CHANGED buffers that may be produced by the component.

Encode

public async Task EncodeFromFilestream()
{
    MMALStandalone standalone = MMALStandalone.Instance;

    using (var stream = File.OpenRead("/home/pi/raw_jpeg_decode.raw"))
    using (var inputCaptureHandler = new InputCaptureHandler(stream))
    using (var outputCaptureHandler = new ImageStreamCaptureHandler("/home/pi/images/", "raw"))
    using (var imgEncoder = new MMALImageEncoder())
    {
        var inputPortConfig = new MMALPortConfig(MMALEncoding.I420, null, 2560, 1920, 0, 0, 0, true, null);
        var outputPortConfig = new MMALPortConfig(MMALEncoding.BMP, MMALEncoding.I420, 2560, 1920, 0, 0, 0, true, null);

        // Create our component pipeline.
        imgEncoder.ConfigureInputPort(inputPortConfig, inputCaptureHandler)
                  .ConfigureOutputPort<FileEncodeOutputPort>(0, outputPortConfig, outputCaptureHandler);

        await standalone.ProcessAsync(imgEncoder);
        Console.WriteLine("Finished");
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    standalone.Cleanup();
}

Decode

public async Task DecodeFromFilestream()
{
    MMALStandalone standalone = MMALStandalone.Instance;

    using (var stream = File.OpenRead("/home/pi/test.jpg"))
    using (var inputCaptureHandler = new InputCaptureHandler(stream))
    using (var outputCaptureHandler = new ImageStreamCaptureHandler("/home/pi/images/", "raw"))
    using (var imgDecoder = new MMALImageDecoder())
    {
        var inputPortConfig = new MMALPortConfig(MMALEncoding.JPEG, MMALEncoding.I420, 2560, 1920, 0, 0, 0, true, null);
        var outputPortConfig = new MMALPortConfig(MMALEncoding.I420, null, 2560, 1920, 0, 0, 0, true, null);

        // Create our component pipeline.
        imgDecoder.ConfigureInputPort(inputPortConfig, inputCaptureHandler)
                  .ConfigureOutputPort<FileEncodeOutputPort>(0, outputPortConfig, outputCaptureHandler);

        await standalone.ProcessAsync(imgDecoder);
        Console.WriteLine("Finished");
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    standalone.Cleanup();
}

Decode -> Encode (and vice-versa)

Available in v0.6.

You can also connect encoder components to decoder components and process as a single operation.

public async Task DecodeThenEncodeImageFromFilestream()
{
    MMALStandalone standalone = MMALStandalone.Instance;

    using (var stream = File.OpenRead("/home/pi/test.bmp"))
    using (var inputCaptureHandler = new InputCaptureHandler(stream))
    using (var outputCaptureHandler = new ImageStreamCaptureHandler("/home/pi/images/", "jpeg"))
    using (var imgDecoder = new MMALImageDecoder())
    using (var imgEncoder = new MMALImageEncoder())
    {
        var decoderInputPortConfig = new MMALPortConfig(MMALEncoding.BMP, MMALEncoding.RGB16, 0, 0, 0, 0, 0, true, null);
        var decoderOutputPortConfig = new MMALPortConfig(MMALEncoding.RGBA, null, 640, 480, 0, 0, 0, true, null);

        var encoderInputPortConfig = new MMALPortConfig(MMALEncoding.RGBA, null, 640, 480, 0, 0, 0, true, null);
        var encoderOutputPortConfig = new MMALPortConfig(MMALEncoding.JPEG, MMALEncoding.I420, 640, 480, 0, 0, 0, true, null);

        // Create our component pipeline.
        imgDecoder.ConfigureInputPort(decoderInputPortConfig, inputCaptureHandler)
                  .ConfigureOutputPort(0, decoderOutputPortConfig, null);

        imgEncoder.ConfigureInputPort(encoderInputPortConfig, imgDecoder.Outputs[0], null)
                  .ConfigureOutputPort<FileEncodeOutputPort>(0, encoderOutputPortConfig, outputCaptureHandler);

        imgDecoder.Outputs[0].ConnectTo(imgEncoder);

        standalone.PrintPipeline(imgDecoder);

        await standalone.ProcessAsync(imgDecoder);

        Console.WriteLine("Finished");
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    standalone.Cleanup();
}

Encode / Decode from Stream - Video

You can also encode and decode video files fed from streams in MMALSharp.

Encode

public async Task EncodeVideoFromFilestream()
{
    MMALStandalone standalone = MMALStandalone.Instance;

    using (var stream = File.OpenRead("/home/pi/videos/decoded_rgb.raw"))
    using (var inputCaptureHandler = new InputCaptureHandler(stream))
    using (var outputCaptureHandler = new VideoStreamCaptureHandler("/home/pi/videos/", "h264"))
    using (var vidEncoder = new MMALVideoEncoder())
    {
        var inputPortConfig = new MMALPortConfig(MMALEncoding.RGB16, null, 1280, 720, 25, 0, 1300000, true, null);
        var outputPortConfig = new MMALPortConfig(MMALEncoding.H264, MMALEncoding.RGB16, 1280, 720, 25, 0, 1300000, true, null);

        // Create our component pipeline.
        vidEncoder.ConfigureInputPort(inputPortConfig, inputCaptureHandler)
            .ConfigureOutputPort<FileEncodeOutputPort>(0, outputPortConfig, outputCaptureHandler);

        await standalone.ProcessAsync(vidEncoder);
        Console.WriteLine("Finished");
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    standalone.Cleanup();
}

Decode

public async Task DecodeVideoFromFilestream()
{
    MMALStandalone standalone = MMALStandalone.Instance;

    using (var stream = File.OpenRead("/home/pi/videos/test.h264"))
    using (var inputCaptureHandler = new InputCaptureHandler(stream))
    using (var outputCaptureHandler = new VideoStreamCaptureHandler("/home/pi/videos/", "raw"))
    using (var vidDecoder = new MMALVideoDecoder())
    {
        var inputPortConfig = new MMALPortConfig(MMALEncoding.H264, null, 1280, 720, 25, 0, 1300000, true, null);
        var outputPortConfig = new MMALPortConfig(MMALEncoding.RGB16, null, 1280, 720, 25, 0, 1300000, true, null);

        // Create our component pipeline.
        vidDecoder.ConfigureInputPort(inputPortConfig, inputCaptureHandler)
            .ConfigureOutputPort<FileEncodeOutputPort>(0, outputPortConfig, outputCaptureHandler);

        await standalone.ProcessAsync(vidDecoder);
        Console.WriteLine("Finished");
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    standalone.Cleanup();
}

Decode -> Encode (and vice-versa)

Available in v0.6.

Again, you can also connect encoder components to decoder components and process as a single operation.

public async Task DecodeThenEncodeVideoFromFilestream()
{
    MMALStandalone standalone = MMALStandalone.Instance;

    using (var stream = File.OpenRead("/home/pi/test.h264"))
    using (var inputCaptureHandler = new InputCaptureHandler(stream))
    using (var outputCaptureHandler = new VideoStreamCaptureHandler("/home/pi/Videos", "h264"))
    using (var imgDecoder = new MMALVideoDecoder())
    using (var imgEncoder = new MMALVideoEncoder())
    {
        var decoderInputConfig = new MMALPortConfig(MMALEncoding.H264, null, 1280, 720, 25, 0, 0, true, null);
        var decoderOutputConfig = new MMALPortConfig(MMALEncoding.I420, null, 1280, 720, 0, 0, 0, true, null);

        var encoderInputConfig = new MMALPortConfig(MMALEncoding.I420, null, 1280, 720, 0, 0, 0, true, null);
        var encoderOutputConfig = new MMALPortConfig(MMALEncoding.H264, MMALEncoding.I420, 1280, 720, 25, 0, 0, true, null);

        imgDecoder.ConfigureInputPort(decoderInputConfig, inputCaptureHandler)
                  .ConfigureOutputPort(0, decoderOutputConfig, null);

        imgEncoder.ConfigureInputPort(encoderInputConfig, imgDecoder.Outputs[0], null)
                  .ConfigureOutputPort<FileEncodeOutputPort>(0, encoderOutputConfig, outputCaptureHandler);
        
        imgDecoder.Outputs[0].ConnectTo(imgEncoder);
        
        await standalone.ProcessAsync(imgDecoder);

        Console.WriteLine("Finished");
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    standalone.Cleanup();
}

Decode -> Splitter -> Encode (and vice-versa)

Available in v0.6.

The flexibility of the MMAL pipeline allows you to also add a splitter component into the mix. You can start with a Decoder component, connect it to a splitter (giving you 4 outputs), and then attach 4 encoders onto each splitter output port.

In the example below, we are assuming that the input video file is H.264 YUV420 encoded with a resolution of 1280 x 720. This file is then decoded, sent to the splitter component and fed to 4 individual encoder components, re-encoding to exactly the same format. You are free to tinker around with the encodings/pixel formats.

public async Task DecodeThenEncodeVideoFromFilestreamWithSplitter()
{
    MMALStandalone standalone = MMALStandalone.Instance;

    using (var stream = File.OpenRead("/home/pi/test.h264"))
    using (var inputCaptureHandler = new InputCaptureHandler(stream))
    using (var outputCaptureHandler = new VideoStreamCaptureHandler("/home/pi/Videos", "h264"))
    using (var outputCaptureHandler2 = new VideoStreamCaptureHandler("/home/pi/Videos", "h264"))
    using (var outputCaptureHandler3 = new VideoStreamCaptureHandler("/home/pi/Videos", "h264"))
    using (var outputCaptureHandler4 = new VideoStreamCaptureHandler("/home/pi/Videos", "h264"))
    using (var splitter = new MMALSplitterComponent())
    using (var imgDecoder = new MMALVideoDecoder())
    using (var imgEncoder = new MMALVideoEncoder())
    using (var imgEncoder2 = new MMALVideoEncoder())
    using (var imgEncoder3 = new MMALVideoEncoder())
    using (var imgEncoder4 = new MMALVideoEncoder())
    {
        var decoderInputConfig = new MMALPortConfig(MMALEncoding.H264, null, 1280, 720, 25, 0, 0, true, null);
        var decoderOutputConfig = new MMALPortConfig(MMALEncoding.I420, null, 1280, 720, 0, 0, 0, true, null);
        
        var splitterInputConfig = new MMALPortConfig(MMALEncoding.I420, null, 0, 0, 25, 0, 0, true, null);
        var splitterOutputConfig = new MMALPortConfig(null, null, 0, 0, 0, 0, 0, true, null);

        var encoderInputConfig = new MMALPortConfig(MMALEncoding.I420, null, 1280, 720, 0, 0, 0, true, null);
        var encoderOutputConfig = new MMALPortConfig(MMALEncoding.H264, MMALEncoding.I420, 1280, 720, 25, 0, 0, true, null);

        imgDecoder.ConfigureInputPort(decoderInputConfig, inputCaptureHandler)
                  .ConfigureOutputPort(0, decoderOutputConfig, null);

        splitter.ConfigureInputPort(splitterInputConfig, null)
                .ConfigureOutputPort(0, splitterOutputConfig, null);

        imgEncoder.ConfigureInputPort(encoderInputConfig, splitter.Outputs[0], null)
                  .ConfigureOutputPort<FileEncodeOutputPort>(0, encoderOutputConfig, outputCaptureHandler);
                  
        imgEncoder2.ConfigureInputPort(encoderInputConfig, splitter.Outputs[1], null)
                  .ConfigureOutputPort<FileEncodeOutputPort>(0, encoderOutputConfig, outputCaptureHandler);
        
        imgEncoder3.ConfigureInputPort(encoderInputConfig, splitter.Outputs[2], null)
                  .ConfigureOutputPort<FileEncodeOutputPort>(0, encoderOutputConfig, outputCaptureHandler);
                  
        imgEncoder4.ConfigureInputPort(encoderInputConfig, splitter.Outputs[3], null)
                  .ConfigureOutputPort<FileEncodeOutputPort>(0, encoderOutputConfig, outputCaptureHandler);
        
        imgDecoder.Outputs[0].ConnectTo(splitter);
        splitter.Outputs[0].ConnectTo(imgEncoder);
        splitter.Outputs[1].ConnectTo(imgEncoder2);
        splitter.Outputs[2].ConnectTo(imgEncoder3);
        splitter.Outputs[3].ConnectTo(imgEncoder4);
        
        await standalone.ProcessAsync(imgDecoder);

        Console.WriteLine("Finished");
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    standalone.Cleanup();
}

Decode -> Splitter -> Resizer -> Encode (and vice-versa)

Available in v0.6.

In addition to the above, we can also introduce a resizer component too.

In the example below, we are assuming that the input video file is H.264 YUV420 encoded with a resolution of 1280 x 720. This file is then decoded, sent to the splitter component and fed to 4 individual encoder components, one of which via a resizer component where we resize the output to 640 x 480. We then re-encode to exactly the same format. You are free to tinker around with the encodings/pixel formats.

public async Task DecodeThenEncodeVideoFromFilestreamWithSplitterAndResizer()
{
    MMALStandalone standalone = MMALStandalone.Instance;

    using (var stream = File.OpenRead("/home/pi/test.h264"))
    using (var inputCaptureHandler = new InputCaptureHandler(stream))
    using (var outputCaptureHandler = new VideoStreamCaptureHandler("/home/pi/Videos", "h264"))
    using (var outputCaptureHandler2 = new VideoStreamCaptureHandler("/home/pi/Videos", "h264"))
    using (var outputCaptureHandler3 = new VideoStreamCaptureHandler("/home/pi/Videos", "h264"))
    using (var outputCaptureHandler4 = new VideoStreamCaptureHandler("/home/pi/Videos", "h264"))
    using (var splitter = new MMALSplitterComponent())
    using (var imgDecoder = new MMALVideoDecoder())
    using (var imgEncoder = new MMALVideoEncoder())
    using (var imgEncoder2 = new MMALVideoEncoder())
    using (var imgEncoder3 = new MMALVideoEncoder())
    using (var imgEncoder4 = new MMALVideoEncoder())
    using (var resizer = new MMALResizerComponent())
    {
        var decoderInputConfig = new MMALPortConfig(MMALEncoding.H264, null, 1280, 720, 25, 0, 0, true, null);
        var decoderOutputConfig = new MMALPortConfig(MMALEncoding.I420, null, 1280, 720, 0, 0, 0, true, null);
        
        var splitterInputConfig = new MMALPortConfig(MMALEncoding.I420, null, 0, 0, 25, 0, 0, true, null);
        var splitterOutputConfig = new MMALPortConfig(null, null, 0, 0, 0, 0, 0, true, null);

        var resizerInputConfig = new MMALPortConfig(MMALEncoding.I420, null, 1280, 720, 25, 0, 0, true, null);
        var resizerOutputConfig = new MMALPortConfig(null, null, 640, 480, 0, 0, 0, true, null);

        var encoderInputConfig = new MMALPortConfig(MMALEncoding.I420, null, 1280, 720, 0, 0, 0, true, null);
        var encoderOutputConfig = new MMALPortConfig(MMALEncoding.H264, MMALEncoding.I420, 1280, 720, 25, 0, 0, true, null);

        imgDecoder.ConfigureInputPort(decoderInputConfig, inputCaptureHandler)
                  .ConfigureOutputPort(0, decoderOutputConfig, null);

        splitter.ConfigureInputPort(splitterInputConfig, null)
                .ConfigureOutputPort(0, splitterOutputConfig, null);

        imgEncoder.ConfigureInputPort(encoderInputConfig, splitter.Outputs[0], null)
                  .ConfigureOutputPort<FileEncodeOutputPort>(0, encoderOutputConfig, outputCaptureHandler);
                  
        imgEncoder2.ConfigureInputPort(encoderInputConfig, splitter.Outputs[1], null)
                  .ConfigureOutputPort<FileEncodeOutputPort>(0, encoderOutputConfig, outputCaptureHandler);
        
        imgEncoder3.ConfigureInputPort(encoderInputConfig, splitter.Outputs[2], null)
                  .ConfigureOutputPort<FileEncodeOutputPort>(0, encoderOutputConfig, outputCaptureHandler);
        
        resizer.ConfigureInputPort(resizerInputConfig, splitter.Outputs[3], null)
               .ConfigureOutputPort(0, resizerOutputConfig, null);
        
        imgEncoder4.ConfigureInputPort(encoderInputConfig, resizer.Outputs[0], null)
                  .ConfigureOutputPort<FileEncodeOutputPort>(0, encoderOutputConfig, outputCaptureHandler);
        
        imgDecoder.Outputs[0].ConnectTo(splitter);
        splitter.Outputs[0].ConnectTo(imgEncoder);
        splitter.Outputs[1].ConnectTo(imgEncoder2);
        splitter.Outputs[2].ConnectTo(imgEncoder3);
        splitter.Outputs[3].ConnectTo(resizer);
        resizer.Outputs[0].ConnectTo(imgEncoder4);
        
        await standalone.ProcessAsync(imgDecoder);

        Console.WriteLine("Finished");
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    standalone.Cleanup();
}

Static render overlay

MMAL allows you to create additional video preview renderers which sit alongside the usual Null Sink or Video renderers shown in previous examples. The purpose of the additional renderers is that they allow you to overlay static content which is shown onto the display your Pi is connected to.

The overlay renderers will only work with unencoded images and they must have one of the following pixel formats:

  • YUV420 (I420)
  • RGB888 (RGB24)
  • RGBA
  • BGR888 (BGR24)
  • BGRA

An easy way to get an unencoded image for use with the overlay renderers is to use the Raw image capture functionality as described in this example, setting the MMALCameraConfig.StillEncoding and MMALCameraConfig.StillSubFormat properties to one of the accepted pixel formats. Once you have got your test frame, follow the below example to overlay your image:

public async Task StaticOverlayExample()
{                        
    MMALCamera cam = MMALCamera.Instance;
    
    PreviewConfiguration previewConfig = new PreviewConfiguration
    {
        FullScreen = false,
        PreviewWindow = new Rectangle(160, 0, 640, 480),
        Layer = 2,
        Opacity = 1
    };

    using (var imgCaptureHandler = new ImageStreamCaptureHandler("/home/pi/images/", "jpg"))
    using (var imgEncoder = new MMALImageEncoder())
    using (var video = new MMALVideoRenderer(previewConfig))
    {                    
        cam.ConfigureCameraSettings();
        video.ConfigureRenderer();
                
        PreviewOverlayConfiguration overlayConfig = new PreviewOverlayConfiguration
        {
            FullScreen = true,
            PreviewWindow = new Rectangle(50, 0, 640, 480),
            Layer = 1,
            Resolution = new Resolution(640, 480),
            Encoding = MMALEncoding.I420,
            Opacity = 255
        };
                
        var overlay = cam.AddOverlay(video, overlayConfig, File.ReadAllBytes("/home/pi/test1.raw"));
        overlay.ConfigureRenderer();
        overlay.UpdateOverlay();
             
        var portConfig = new MMALPortConfig(MMALEncoding.JPEG, MMALEncoding.I420, 90);

        //Create our component pipeline.  
        imgEncoder.ConfigureOutputPort(portConfig, imgCaptureHandler);
                
        cam.Camera.StillPort.ConnectTo(imgEncoder);
        cam.Camera.PreviewPort.ConnectTo(video);
                
        cam.PrintPipeline();
                
        await cam.ProcessAsync(cam.Camera.StillPort);        
    }
    
    // Only call when you no longer require the camera, i.e. on app shutdown.
    cam.Cleanup();
}

In this example, we are using an unencoded YUV420 image and configuring the renderer using the settings in overlayConfig.

FFmpeg - RTMP streaming

public async Task FFmpegRTMPStreaming()
{                        
    MMALCamera cam = MMALCamera.Instance;

    // An RTMP server needs to be listening on the address specified in the capture handler. I have used the Nginx RTMP module for testing.    
    using (var ffCaptureHandler = FFmpegCaptureHandler.RTMPStreamer("mystream", "rtmp://192.168.1.91:6767/live"))
    using (var vidEncoder = new MMALVideoEncoder())
    using (var renderer = new MMALVideoRenderer())
    {
        cam.ConfigureCameraSettings(); 

        var portConfig = new MMALPortConfig(MMALEncoding.H264, MMALEncoding.I420, 10, MMALVideoEncoder.MaxBitrateLevel4, null);

        // Create our component pipeline. Here we are using the H.264 standard with a YUV420 pixel format. The video will be taken at 25Mb/s.
        vidEncoder.ConfigureOutputPort(portConfig, ffCaptureHandler);

        cam.Camera.VideoPort.ConnectTo(vidEncoder);
        cam.Camera.PreviewPort.ConnectTo(renderer);
                                
        // Camera warm up time
        await Task.Delay(2000);

        var cts = new CancellationTokenSource(TimeSpan.FromMinutes(3));
          
        // Take video for 3 minutes.
        await cam.ProcessAsync(cam.Camera.VideoPort, cts.Token);
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    cam.Cleanup();
}

Note:

If you intend on using the YouTube live streaming service, you will need to create the below method to return your own FFmpegCaptureHandler. You should replace the internal FFmpegCaptureHandler.RTMPStreamer seen in the example above with your custom method. The reason for this is YouTube streaming requires your RTMP stream to contain an audio input or otherwise it won't work. Internally, our RTMP streaming method does not include an audio stream, and at the current time we don't intend on changing it for this specific purpose.

public static FFmpegCaptureHandler RTMPStreamerWithAudio(string streamName, string streamUrl)
            => new FFmpegCaptureHandler($"-re -ar 44100 -ac 2 -acodec pcm_s16le -f s16le -ac 2 -i /dev/zero -f h264 -i - -vcodec copy -acodec aac -ab 128k -g 50 -strict experimental -f flv -metadata streamName={streamName} {streamUrl}");

Please see here which discusses the issue in-depth.

FFmpeg - Raw video convert

This is a useful capture mode as it will push the elementary H.264 stream into an AVI container which can be opened by media players such as VLC.

public async Task FFmpegRawVideoConvert()
{                        
    MMALCamera cam = MMALCamera.Instance;

    using (var ffCaptureHandler = FFmpegCaptureHandler.RawVideoToAvi("/home/pi/videos/", "testing1234"))
    using (var vidEncoder = new MMALVideoEncoder())
    using (var renderer = new MMALVideoRenderer())
    {
        cam.ConfigureCameraSettings(); 

        var portConfig = new MMALPortConfig(MMALEncoding.H264, MMALEncoding.I420, 10, MMALVideoEncoder.MaxBitrateLevel4, null);

        // Create our component pipeline. Here we are using the H.264 standard with a YUV420 pixel format. The video will be taken at 25Mb/s.
        vidEncoder.ConfigureOutputPort(portConfig, ffCaptureHandler);

        cam.Camera.VideoPort.ConnectTo(vidEncoder);
        cam.Camera.PreviewPort.ConnectTo(renderer);
                          
        // Camera warm up time
        await Task.Delay(2000);

        var cts = new CancellationTokenSource(TimeSpan.FromMinutes(3));
                
        // Take video for 3 minutes.
        await cam.ProcessAsync(cam.Camera.VideoPort, cts.Token);
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    cam.Cleanup();
}

FFmpeg - Images to video

This example will push all images processed by an image capture handler into a playable video.

public async Task FFmpegImagesToVideo()
{                        
    MMALCamera cam = MMALCamera.Instance;
    
    // This example will take an image every 10 seconds for 4 hours
    using (var imgCaptureHandler = new ImageStreamCaptureHandler("/home/pi/images/", "jpg"))
    {
        var cts = new CancellationTokenSource(TimeSpan.FromHours(4));

        var tl = new Timelapse { Mode = TimelapseMode.Second, CancellationToken = cts.Token, Value = 10 };
        await cam.TakePictureTimelapse(imgCaptureHandler, MMALEncoding.JPEG, MMALEncoding.I420, tl);

        // Process all images captured into a video at 2fps.
        imgCaptureHandler.ImagesToVideo("/home/pi/images/", 2);
    }
    
    // Only call when you no longer require the camera, i.e. on app shutdown.
    cam.Cleanup();
}

Camera modules

OmniVision OV5647

Sony IMX219

Sony IMX477

Install guide

MMAL

What is MMAL?

Colour Models

MMALSharp

Introduction

Installation

Configuration

Handling data

Unit tests

FAQ

Examples

Basic Examples

Advanced Examples

CCTV and Motion Detection

Image Processing

Websocket Streaming - Bobcat

Clone this wiki locally