-
Notifications
You must be signed in to change notification settings - Fork 33
CCTV and Motion Detection
- v0.7 Alpha
- v0.6 (Current)
- Motion Detection Basics
- CCTV (Security Cameras)
- Advanced Usage
Frame differencing is a common motion-detection technique whereby a test frame (sometimes called the "background frame") is compared against new frames (or "current frame") for changes exceeding various thresholds. The MMALSharp library has new APIs and classes that let you configure motion detection behavior, including callbacks to run custom code when motion is detected.
There are different strategies to detect differences between frames. The provided implementation combines two techniques which help reject sensor noise and small localized motion (such as an insect, or even a small pet).
At the most basic level, the algorithm compares individual pixels. This is called "RGB summing" because the red, green, and blue values are added together for each pixel in both images. If the difference between the test frame and the new frame exceeds a threshold, the pixel is considered changed. The image is subdivided into a grid of smaller rectangles called cells. The size of each cell and the number of pixels in the cell depends on the image resolution. There is a second threshold which defines the percentage of pixels in the cell which must change for the entire cell to be considered changed. This is how sensor noise and other minor changes are discarded. Finally, there is a third threshold, which is the number of cells across the entire image that must register changes in order to signal that motion detection has occurred. This is how real but small and unimportant motion is ignored (insects, pets, and distant background movement, for example). All of these thresholds are configurable.
Typically motion detection doesn't require or benefit from high resolution. 640 x 480 should be adequate, although you should always feed raw RGB24, RGB32, or RGBA images into the system. Image artifacts from lossy compression algorithms like h.264 will be mistaken for motion and the RGB summing algorithm is not compatible with the YUV pixel format. At 640 x 480 x RGB24, a Raspberry Pi 4B can easily process full-motion video using the provided algorithms (an improvement over v0.6 which could only process about 5 frames per second on the same hardware).
The new FrameBufferCaptureHandler class provides management and control of motion detection. The following example demonstrates the most basic possible motion detection. This does nothing but write messages to the console when motion is detected. Later we'll see more complete examples that capture video and snapshots.
public async Task SimpleMotionDetection(int totalSeconds)
{
// Assumes the camera has been configured.
var cam = MMALCamera.Instance;
using (var motionCaptureHandler = new FrameBufferCaptureHandler())
using (var resizer = new MMALIspComponent())
{
// The ISP resizer is used to output a small (640x480) image to ensure high performance. As described in the
// wiki, frame difference motion detection only works reliably on uncompressed, unencoded raw RGB data. The
// resizer outputs this raw frame data directly into the motion detection handler.
resizer.ConfigureInputPort(new MMALPortConfig(MMALEncoding.OPAQUE, MMALEncoding.I420), cam.Camera.VideoPort, null);
resizer.ConfigureOutputPort<VideoPort>(0, new MMALPortConfig(MMALEncoding.RGB24, MMALEncoding.RGB24, width: 640, height: 480), motionCaptureHandler);
cam.Camera.VideoPort.ConnectTo(resizer);
// Camera warm-up.
await Task.Delay(2000);
// We'll use the default settings for this example.
var motionConfig = new MotionConfig(algorithm: new MotionAlgorithmRGBDiff());
// Duration of the motion-detection operation.
var stoppingToken = new CancellationTokenSource(TimeSpan.FromSeconds(totalSeconds));
Console.WriteLine($"Detecting motion for {totalSeconds} seconds.");
await cam.WithMotionDetection(
motionCaptureHandler,
motionConfig,
// This callback will be invoked when motion has been detected.
async () =>
{
// When motion is detected, temporarily disable notifications
motionCaptureHandler.DisableMotionDetection();
// Wait 2 seconds
Console.WriteLine($"\n {DateTime.Now:hh\\:mm\\:ss} Motion detected, disabling detection for 2 seconds.");
await Task.Delay(2000, stoppingToken.Token);
// Re-enable motion detection
if(!stoppingToken.IsCancellationRequested)
{
Console.WriteLine($" {DateTime.Now:hh\\:mm\\:ss} ...motion detection re-enabled.");
motionCaptureHandler.EnableMotionDetection();
}
})
.ProcessAsync(cam.Camera.VideoPort, stoppingToken.Token);
}
cam.Cleanup();
}The WithMotionDetection method configures the camera processing loop for motion detection by identifying the FrameBufferCaptureHandler responsible for motion detection, the MotionConfig defining the applicable settings, and an asynchronous callback which is invoked when motion is detected.
IMPORTANT: It is your responsibility to ensure all exceptions are handled inside your callback.
The callback is an event handler, which means it is an async void delegate. Event handlers are the only scenario in .NET applications where async void is an acceptable method signature (versus the common async Task signature). Since it returns void instead of Task, there is no enclosing method which can intercept an exception, and unhandled exceptions will immediately terminate the process.
Motion detection commonly requires ignoring areas of the camera view where real or apparent motion may occur that is not of interest. The library allows you to configure a mask bitmap to define areas to be ignored.
Masking is especially useful for outdoor scenes where "background" motion like trees, clouds, or passing vehicular traffic may trigger unwanted events. Masking can also be helpful indoors where changes like reflections in a picture frame, movement on a television screen, or even blinking LEDs on electronic devices may be mistaken as motion.
The mask bitmap must be the same size and color depth as the motion detection frames, and the file format should be either BMP or PNG format. The library can also load a JPG mask file, but this is not recommended as compression artifacts may produce inaccuracies.
Fully-black pixels in the mask will be ignored -- they will always be treated as if no motion has occurred. Thus, the easiest way to create a mask is to capture a still picture (for example, using the raspistill utility with a -e BMP or -e PNG encoding switch) and load that into any image editor to blank out the unwanted regions.
The mask is specified as an optional pathname argument to the MotionConfig constructor:
var motionConfig = new MotionConfig(
algorithm: new MotionAlgorithmRGBDiff(),
maskBitmap: "/home/pi/images/motionmask.bmp"
);An exception will be thrown if the mask cannot be found, or if the resolution or color-depth does not match the motion detection image configuration (in these examples, that is always 640 x 480 x RGB24).
Testing has not shown any discernable changes to performance when a mask is used.
Motion detection based on frame differencing algorithms compares a test frame to newly received frames. The first full frame captured by the camera is stored as the test frame. To help compensate for gradual changes in the scene most commonly caused by lighting changes and shadows, the library is able to periodically update the test frame with a new image.
These are values you will likely need to tune for the specific scene your camera is viewing. Although you can adjust this through trial and error, it may be easier to view the algorithm output in real-time. Refer to the streaming visualisation topic later in this area of the documentation.
Two optional arguments to the MotionConfig constructor controls how this works. Both values default to 3 seconds:
var motionConfig = new MotionConfig(
algorithm: new MotionAlgorithmRGBDiff(),
testFrameInterval: TimeSpan.FromSeconds(3),
testFrameCooldown: TimeSpan.FromSeconds(3)
);The testFrameInterval defines how often the test frame is updated, and testFrameCooldown defines how long the scene must be "quiet" (no motion detected) before a test frame is updated. The cooldown period is checked after the interval passes, so the default values of 3 seconds means it will actually update every 6 seconds at a minimum, and possibly longer if there is ongoing motion.
Note that the cooldown is relative to triggered motion. If the scene contains minor motion that was not sufficient to trigger a motion detection event, it's possible that the new test frame will capture a moving object. If you see this happening, simply increase the intervals, the default intervals are somewhat aggressively short.
The library supports different motion detection algorithms, but currently only one algorithm is built in -- RGB summing (also called RGB differencing). While the core motion detection system is based on frame differencing, RGB differencing is based on changes at the pixel level. Because camera image sensors are naturally "noisy", and also to help reject other sources of minor, uninteresting motion, the algorithm also requires larger-scale changes at the "cell" level. Cells are an arbitrarily-sized grid applied to the image data.
The MotionConfig constructor requires a motionAlgorithm object, and the built-in MotionAlgorithmRGBDiff constructor accepts three optional arguments to control sensitivity:
var motionConfig = new MotionConfig(
algorithm: new MotionAlgorithmRGBDiff(
rgbThreshold: 200,
cellPixelPercentage: 50,
cellCountThreshold: 20
));The settings shown above are the defaults.
The rgbThreshold setting controls change-detection sensitivity at the individual pixel level. The maximum value is 255 + 255 + 255 which is 765. Since the per-pixel RGB difference algorithm compares test frame pixels to new frame pixels, a value of 765 would only indicate a change when a fully-black pixel (RGB 0,0,0) switched to full-white (RGB 255,255,255) or vice-versa, so clearly much lower values are more useful. This sensitivity setting helps reject minor lighting changes and the like.
Each image frame is subdivided into a grid of "cells" based on the image resolution. The library automatically selects the grid size. The recommended resolution for motion detection is 640 x 480 which uses a 32 x 32 grid for a total of 1024 cells. This means each cell represents 20 x 15 pixels, or 300 pixels. (The number of cells varies by resolution, but most are around 800 to 1000 -- refer to the table at the end of this section.)
Each cell tracks the number of pixels that changed within the cell. When that count reaches the cellPixelPercentage value, the entire cell is considered to have changed. If the count is below that percentage, and the cell is considered unchanged, even if some pixels within the cell have changed. So given a 640 x 480 image using 300-pixel-count cells, the default 50% threshold means 150 pixels or more must change (exceed the rgbThreshold) within that cell to trigger a change for the entire cell. This setting helps reject very small sources of motion such as insects or a falling leaf.
Finally, motion detection events are triggered by the total number of cells which have changed using the two processes described above. The cellCountThreshold defines the minimum number of cells across the entire image that must change before the motion detection callback is invoked. This helps reject somewhat larger sources of motion such as small pets or even a television screen within view (although that's more easily ignored with a mask bitmap).
These are values you may want to tune for the types of motion you wish to detect. Although you can adjust this through trial and error, it may be easier to view the algorithm output in real-time. Refer to the streaming visualisation topic later in this area of the documentation.
The OnDetect delegate that you provide to respond to motion detection events can perform any action you desire: record video, take still pictures (or both, as shown below), write to log files, send emails or mobile phone messages, and so on.
The examples in this wiki call DisableMotionDetection while these activities are performed, then later call EnableMotionDetection to re-enable notification. When called in this way, disable stops everything inside the motion detection code, and the system is reset when it is re-enabled. This means the test frame update logic is interrupted, and a new test frame is stored as soon as the system is re-enabled. This is fine for simple demos, but in a more realistic usage, and particularly for CCTV where the system is running for long periods of time, you want that test frame logic to continue running in the background while your code responds to the motion event. For this reason, DisableMotionDetection accepts a bool argument, disableCallbackOnly, which allows the algorithm to continue working without invoking your delegate again.
Another alternative is to design your delegate to be tolerant of multiple frequent invocations (likely one per frame, when motion is being detected), rather than disabling motion detection at all. This is probably how a more sophisticated CCTV application would be designed. For example, you might track the duration between invocations and use a CancellationToken timeout to periodically test for the end of a motion event. This type of design also allows your program to continue operating as if motion is still detected when there are momentary interruptions in actual detection events. The details will depend heavily on your specific needs.
Because video files are very large, it is impractical to simply record everything the camera sees. CCTV systems usually save short video clips when motion is detected, and the better systems continuously buffer a certain amount of video so that the stored video clip includes a time period before motion was detected. The MMALSharp library has a component called the CircularBufferCaptureHandler which is capable of doing exactly this.
The following example expands on the basic motion detection example by adding a splitter, a video encoder, and the new handler to output encoded h.264 video clips.
public async Task RecordMotion(int totalSeconds, int recordSeconds)
{
// Assumes the camera has been configured.
var cam = MMALCamera.Instance;
// h.264 requires key frames for the circular buffer capture handler.
MMALCameraConfig.InlineHeaders = true;
using (var videoCaptureHandler = new CircularBufferCaptureHandler(4000000, "/home/pi/videos/detections", "h264"));
using (var motionCaptureHandler = new FrameBufferCaptureHandler())
using (var resizer = new MMALIspComponent())
using (var splitter = new MMALSplitterComponent())
using (var videoEncoder = new MMALVideoEncoder())
{
splitter.ConfigureInputPort(new MMALPortConfig(MMALEncoding.OPAQUE, MMALEncoding.I420), cam.Camera.VideoPort, null);
videoEncoder.ConfigureOutputPort(new MMALPortConfig(MMALEncoding.H264, MMALEncoding.I420, 0, MMALVideoEncoder.MaxBitrateLevel4, null), videoCaptureHandler);
// As with the basic example, the resizer sends 640 x 480 raw frames to the motion detection handler.
resizer.ConfigureOutputPort<VideoPort>(0, new MMALPortConfig(MMALEncoding.RGB24, MMALEncoding.RGB24, width: 640, height: 480), motionCaptureHandler);
cam.Camera.VideoPort.ConnectTo(splitter);
splitter.Outputs[0].ConnectTo(resizer);
splitter.Outputs[1].ConnectTo(videoEncoder);
// Camera warm-up.
await Task.Delay(2000);
// We'll use the default settings for this example.
var motionConfig = new MotionConfig(algorithm: new MotionAlgorithmRGBDiff());
// Duration of the motion-detection operation.
var stoppingToken = new CancellationTokenSource(TimeSpan.FromSeconds(totalSeconds));
Console.WriteLine($"Detecting motion for {totalSeconds} seconds.");
await cam.WithMotionDetection(
motionCaptureHandler,
motionConfig,
// This callback will be invoked when motion has been detected.
async () =>
{
// When motion is detected, temporarily disable notifications
motionCaptureHandler.DisableMotionDetection();
Console.WriteLine($"\n {DateTime.Now:hh\\:mm\\:ss} Motion detected, recording for {recordSeconds} seconds.");
// When the recording period expires, stop recording and re-enable capture
var stopRecording = new CancellationTokenSource();
stopRecording.Token.Register(() =>
{
Console.WriteLine($" {DateTime.Now:hh\\:mm\\:ss} ...recording stopped.");
motionCaptureHandler.EnableMotionDetection();
// Calling split will close the h.264 file stream and open another file to
// store new circular buffer data while we wait for another motion event.
videoCaptureHandler.StopRecording();
videoCaptureHandler.Split();
});
// Start the recording countdown
stopRecording.CancelAfter(recordSeconds * 1000);
// Record until the duration passes or the overall motion detection token expires
await Task.WhenAny(
// Calling StartRecording saves the contents of the circular buffer, then begins appending new
// video frames to the buffer until StopRecording is called. The first argument is an optional
// initialization Action, which in this case ensures the h.264 stream emits an IFrame.
videoCaptureHandler.StartRecording(videoEncoder.RequestIFrame, stopRecording.Token),
stoppingToken.Token.AsTask()
);
// If the awaiter above exited because the overall stoppingToken
// has expired, ensure we also terminate the ongoing recording.
if(!stopRecording.IsCancellationRequested) stopRecording.Cancel();
})
.ProcessAsync(cam.Camera.VideoPort, stoppingToken.Token);
}
cam.Cleanup();
}Content TBD
Content TBD
Content TBD
Although we strongly recommend using 640 x 480 x RGB24 for motion-detection, the algorithms should theoretically work with raw frame data of any resolution. (Throughput has been measured to decrease linearlly as the resolution increases.) The following tables indicate the cell dimensions applied to the various available camera modules and resolutions.
v1 camera (OV5647)
Mode Resolution Cells Total Pixels
1 1920 x 1080 30 x 30 900 64 x 36
2,3 2592 x 1944 36 x 36 1296 72 x 54
4 1296 x 972 27 x 27 729 36 x 48
5 1296 x 730 72 x 10 720 18 x 73
6,7 640 x 480 32 x 32 1024 20 x 15
v2 camera (IMX219)
Mode Resolution Cells Total Pixels
1 1920 x 1080 30 x 30 900 64 x 36
2,3 3280 x 2464 40 x 22 880 82 x 112
4 1640 x 1232 40 x 22 880 41 x 56
5 1640 x 922 40 x 23 920 41 x 40.09 (see below)
6 1280 x 720 20 x 36 720 64 x 20
7 640 x 480 32 x 32 1024 20 x 15
HQ camera (IMX477)
Mode Resolution Cells Total Pixels
1 2028 x 1080 26 x 36 936 78 x 30
2 2028 x 1520 26 x 38 988 78 x 40
3 4056 x 3040 26 x 32 832 156 x 95
4 1012 x 760 44 x 19 836 23 x 40
📌 The v2 1640 x 922 resolution (mode 5) does not have a useful vertical-axis divisor. 23 vertical cells yields a cell size of 41 x 40.09 pixels. This means each cell will ignore the right-most column of pixels in the cell.
The cells are also used to parallel-process the image frames, which is why they have similar total cell counts. Around 800 cells seems to be the optimal number for parallel processing on the Raspberry Pi. The specific cell counts are then chosen to divide evenly into the various image resolutions.