Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

on-NVR video analytics #30

Open
scottlamb opened this issue Mar 9, 2018 · 2 comments
Open

on-NVR video analytics #30

scottlamb opened this issue Mar 9, 2018 · 2 comments
Labels
enhancement rust Rust backend work required schema change Requires a database schema change (see new-schema branch)
Milestone
1.0?

Comments

@scottlamb
Copy link
Owner

scottlamb commented Mar 9, 2018 •
edited
Loading

IMHO, this is superior to on-camera motion detection (#29) because from what I've seen on-camera motion detection kind of sucks:

  • it has a bunch of false negatives and positives; there at least exist algorithms I could try on the NVR that are somewhat better
  • it has completely different options between camera brands
  • it's very tedious to tune because you can't just save a bunch of video segments and try out different options on them
  • it's all a closed-source, mostly undocumented black box

on the other hand, on-NVR has more hardware needs (details TBD but significant work to do which scales with the number of cameras) and is much more complex to implement:

  • needs more schema work as noted in "signals" schema (for motion detection and such) #28 to track config (algorithm, parameters, mask), progress (last segment handled, reference image), and perhaps output (if more detail is desired than just the on/off/unknown signal; maybe a number indicating how much motion is in each grid square or something)
  • needs a driver process that feeds it segments as they become available and probably skips some if it falls too far behind
  • needs H.264/H.265 decoding—not too hard to use ffmpeg for this but might want to take advantage of hardware decoding
  • needs some algorithm development, and there are a lot of possible algorithms (classical and machine learning!) out there with different pros and cons
  • to make a decent algorithm testbed, needs H.264/H.265 encoding so you can create a video that shows you the reference image and motion detection mask alongside the source image
  • might require using fancier instructions/hardware (SIMD, GPU, DSP, ISP, and/or TPU) for decent performance
  • might want to support offloading to another machine via RPC boundary
@scottlamb scottlamb added this to the 1.0? milestone Mar 9, 2018
@scottlamb scottlamb added schema change Requires a database schema change (see new-schema branch) and removed milestone-1.0? labels Mar 9, 2018
@scottlamb
Copy link
Owner Author

Here's an interesting discussion to follow about using machine learning-based object detection with inexpensive NPUs such as the Coral USB Accelerator: motioneye-project/motioneyeos#1505

@scottlamb scottlamb added the rust Rust backend work required label Jul 10, 2019
scottlamb added a commit that referenced this issue Mar 28, 2020
@scottlamb
add a preliminary schema for object detection (#30)
c6fe1b6
@scottlamb scottlamb changed the title on-NVR motion detection on-NVR video analytics Apr 14, 2020
scottlamb added a commit that referenced this issue Apr 14, 2020
@scottlamb
first step toward object detection (#30)
3ed397b 
When compiled with cargo build --features=analytics and enabled via
moonfire-nvr run --object-detection, this runs object detection on every
sub stream frame through an Edge TPU (a Coral USB accelerator) and logs
the result.

This is a very small step toward a working system. It doesn't actually
record the result in the database or send it out on the live stream yet.
It doesn't support running object detection at a lower frame rate than
the sub streams come in at either. To address those problems, I need to
do some refactoring. Currently moonfire_db::writer::Writer::Write is the
only place that knows the duration of the frame it's about to flush,
before it gets added to the index or sent out on the live stream. I
don't want to do the detection from there; I'd prefer the moonfire_nvr
crate. So I either need to introduce an analytics callback or move a
bunch of that logic to the other crate.

Once I do that, I need to add database support (although I have some
experiments for that in moonfire-playground) and API support, then some
kind of useful frontend.

Note edgetpu.tflite is taken from the Apache 2.0-licensed
https://github.com/google-coral/edgetpu,
test_data/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite. The
following page says it's fine to include Apache 2.0 stuff in GPLv3
projects:
https://www.apache.org/licenses/GPL-compatibility.html
scottlamb added a commit to scottlamb/moonfire-playground that referenced this issue Apr 14, 2020
@scottlamb
mass object detection
9ff4646 
This is an experiment for scottlamb/moonfire-nvr#30. I'm reasonably
happy with the schema for representing detected objects.
@scottlamb scottlamb mentioned this issue May 24, 2020
Open
@scottlamb scottlamb mentioned this issue Mar 9, 2021
Closed
@xnorpx
Copy link

xnorpx commented Jan 2, 2024

I wrote this for my blue iris server, https://github.com/xnorpx/blue-candle but it's far away from realtime so it works best with some simple classic motion detection and then run yolo to determine the actual object.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement rust Rust backend work required schema change Requires a database schema change (see new-schema branch)
Projects
None yet
Milestone
1.0?
Development

No branches or pull requests
2 participants
@xnorpx @scottlamb