|
| 1 | +# Video Senders (Qubes RPC Services) |
| 2 | + |
| 3 | +## -q (quiet) |
| 4 | +### gst-launch-1.0 needs to be quiet because the debug messages are printed on standard output instead of standard error |
| 5 | +- This means the debug info will become part of the video output upon being sent to the `fdsink` element which is unwanted behavior that results in video artifacts |
| 6 | + |
| 7 | +## queue |
| 8 | +### Force push mode scheduling which is better for a constant stream of data |
| 9 | +- https://gstreamer.freedesktop.org/documentation/additional/design/scheduling.html |
| 10 | + |
| 11 | +## format=I420 |
| 12 | +### This pixel format was chosen for two reasons |
| 13 | +1. I420 seemed to be the default output format for a lot of elements and was always listed at the top of pixel format lists in the GStreamer documentation as well as over places such as on the official FOURCC website which defines these pixel formats |
| 14 | + - So, presumably I420 is the most battle-hardened format which is least likely to contain bugs |
| 15 | + - GStreamer documentation mentions to also prefer I420 over YV12 which are the same but with the V and U planes switched around |
| 16 | +2. I420 is directly compatible with many GStreamer elements including `v4l2sink` without needing a leading `videoconvert` (probably followed by a `tee`) |
| 17 | + - This greatly improves performance and security |
| 18 | +- https://gstreamer.freedesktop.org/documentation/additional/design/mediatype-video-raw.html |
| 19 | +- https://gstreamer.freedesktop.org/documentation/application-development/basics/elements.html |
| 20 | +- https://www.fourcc.org/yuv.php |
| 21 | + |
| 22 | +## use-damage=false in `qvc.ScreenShare` |
| 23 | +### XDamage causes `ximagesrc` to send out small updates about what parts of the screen has changed as opposed to just sending the whole screen |
| 24 | +- This may be preferable on a network because of the decrease in bandwidth and latency but otherwise it just results in very high CPU usage and doesn't fit our use case |
| 25 | + |
| 26 | + |
| 27 | +## BGRx -> I420 pixel format `videoconvert` in `qvc.ScreenShare` |
| 28 | +### `ximagesrc` only outputs in BGRx so it must be converted to I420 which is a supported input format for `v4l2sink` (on the `receiver.sh` side) |
| 29 | +- This video conversion is done on the side of the sending machine as to ensure the attack surface of the recipient stays as small as possible |
| 30 | + |
| 31 | +# Video Receiver (`receiver.sh`) |
| 32 | + |
| 33 | +## capsfilter |
| 34 | +### This is used to limit our attack surface to the given capabilities |
| 35 | +- All the capabilities after the colorimetry are technically unnecessary for this to be functional but are used to limit our attack surface |
| 36 | +- This filter accounts for all the capabilities possible on a raw video stream according to the below documentation |
| 37 | +- https://gstreamer.freedesktop.org/documentation/coreelements/capsfilter.html |
| 38 | +- https://gstreamer.freedesktop.org/documentation/additional/design/mediatype-video-raw.html |
| 39 | +- https://gstreamer.freedesktop.org/documentation/application-development/basics/pads.html#what-capabilities-are-used-for |
| 40 | + |
| 41 | +## colorimetry=2:4:7:1 |
| 42 | +### This is the default colorimetry format for I420 and the only one that works with it without having to specify a `chroma-site` |
| 43 | +- Having `chroma-site` set to `none` reduces our attack surface and likely improves our performance as well |
| 44 | + |
| 45 | +## use-sink-caps=true |
| 46 | +### Use the capabilities defined by the previous element, in this case, what is explicitly defined by the capsfilter |
| 47 | + |
| 48 | +## sync=false |
| 49 | +### Disable syncing video to the system clock |
| 50 | +- This fixes the frame jittering/lagging that happens when passing raw video between machines |
| 51 | +- `sync=false` is the default on the `fdsink` sink on the video sender |
| 52 | + - Tried making both sender and receiver `sync=true` but that resulted in jittering again |
| 53 | + - Also tried making only the `fdsink` sink on the sender `sync=true` and the `v4l2sink` sink on the receiver `sync=false` but that resulted in higher CPU usage and greater video latency |
| 54 | +- Some times it takes some time to start jittering/lagging depending on what the video source is but it will happen |
| 55 | +- I think it's because the two VMs are on slightly different clocks so trying to synchronize to that doesn't work out very well |
| 56 | + - This fix isn't necessary when passing raw video directly between file descriptors on the same machine |
| 57 | +- This may be better fixed by passing timestamp information through the raw video itself |
| 58 | + - However, it appears to be that raw video is incapable of holding timestamp information |
| 59 | + - Timestamping is done on raw video in the GStreamer pipeline but once it leaves there (through `fdsink`) any timestamping information may be void (unsure) |
| 60 | + - Although, research has found a few somewhat hacky solutions for FFmpeg that allowed for timestamping on raw video; perhaps GStreamer has something similar |
| 61 | +- Or maybe we could synchronize the clocks of the machines better thus allowing us to remove `sync=false` |
| 62 | + - Or even just synchronize the clocks that the GStreamer processes see on each machine right before running them |
| 63 | + - Perhaps using the `datefudge` or `faketime` command which uses `LD_PRELOAD` to manipulate the system time for a given command |
0 commit comments