Skip to content

Commit

Permalink
update readme, closes #3
Browse files Browse the repository at this point in the history
  • Loading branch information
floe authored Jan 12, 2022
1 parent daeae11 commit 4d428aa
Showing 1 changed file with 24 additions and 14 deletions.
38 changes: 24 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,44 +1,54 @@
# SurfaceCast

Sends background-subtracted depth camera video via ![GStreamer](https://gstreamer.freedesktop.org/) (left: raw video, center: rectified, right: background removed).
SurfaceCast sends background-subtracted depth camera video via ![GStreamer](https://gstreamer.freedesktop.org/) (left: raw video, center: rectified, right: background removed).

![realsense example](assets/demo.jpg)

This tool records live video of a flat surface with a depth camera, automatically detects the background plane, subtracts that from the video, and sends the result to a GStreamer pipeline. Everything that is part of the background within a configurable distance will turn bright green so it can be used with other GStreamer filters, e.g. using `alpha method=green` and `videomix`. Optionally, an arbitrary quadrilateral area in the raw image can be extracted and rectified before streaming (e.g. a projection screen).
This tool records live video of a flat surface with a depth camera, automatically detects the background plane, subtracts it from the video, and sends the result to a GStreamer pipeline. Everything that is part of the background within a configurable distance will turn bright green so it can be used with other GStreamer filters, e.g. using `alpha method=green` and `videomix`. Optionally, an arbitrary quadrilateral area in the raw image can be extracted and rectified before streaming (e.g. a projection screen).

[![SurfaceStreams Video](assets/anim.gif)](https://www.youtube.com/watch?v=Qe1BROtGyzI "SurfaceStreams Video")

### Usage

By default, all executables will run the GStreamer pipeline `videoconvert ! fpsdisplaysink` to provide a debug view. If you want any other pipeline, pass it as a single commandline parameter, e.g. `./realsense 0 "jpegenc ! rtpgstpay ! udpsink"` to stream the compressed view over a network (first parameter is camera number). In the debug view, the following commands are available:
Example 1 - debug/config view of plain webcam: `./surfacecast v4l2 /dev/video0`

Example 2 - stream Realsense to ![virtual camera device](https://github.com/umlaeute/v4l2loopback): `./surfacecast realsense 0 "videoconvert ! video/x-raw,format=RGB,width=1280,height=720 ! v4l2sink device=/dev/video20"` (mind the quotes around the GStreamer pipeline)

Example 3 - simple network stream of Kinect Azure: `./surfacecast k4a 0 "jpegenc ! rtpjpegpay ! udpsink host=..."`

By default, 1280x720 RGB video data will be sent to the GStreamer pipeline `videoconvert ! fpsdisplaysink` to provide a debug view. If you want any other pipeline, pass it as a single quoted commandline parameter. In the debug view, the following hotkeys are available:

* `space` - reset the perspective transformation
* `mouse` - create new perspective transformation
* click the 4 corners of the desired quadrilateral area in succession
* first click defines top-left corner, rest in clockwise order
* transformation will be saved as `perspective.xml` in current directory
* `p` - re-run the RANSAC plane detection
* `f` - toggle filtering of the background plane
* `+` - increase the tolerance wrt background by 0.2cm
* `-` - decrease the tolerance wrt background by 0.2cm
* `q` - exit program

After the perspective calibration has been completed, all current parameters (tolerance, perspective transform, plane parameters) are saved to `config.xml` in current directory; if this file is present at startup, parameters are re-initialized with saved values.

### Supported/tested devices:

* Intel Realsense D415
* Microsoft Kinect Azure
* Samsung SUR40
* Generic Video4Linux2 camera (without background subtraction)
* Microsoft Kinect v2 (deprecated)
* Samsung SUR40 (with brightness-based background subtraction)
* Intel Realsense D415/D435
* Microsoft Kinect Azure
* Virtual Camera

### Build requirements:

### Library requirements:
Reference platform: Ubuntu 20.04.

* gstreamer-1.16 (libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev)
* opencv-4.2.0 (libopencv-dev libeigen3-dev)
* libk4a-1.4.1
* librealsense2-2.50 (https://github.com/IntelRealSense/librealsense)
* libfreenect-0.2-gstreamer (deprecated) (https://github.com/floe/libfreenect2/tree/gstreamer)
* General (V4L2, SUR40, virtual camera):
* gstreamer-1.16 (`libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev`)
* opencv-4.2.0 (`libopencv-dev libeigen3-dev`)
* Kinect Azure:
* libk4a-1.4.1 (https://github.com/microsoft/Azure-Kinect-Sensor-SDK)
* Realsense:
* librealsense2-2.50 (https://github.com/IntelRealSense/librealsense)

### License

Expand Down

0 comments on commit 4d428aa

Please sign in to comment.