Development ~ Room for Improvement

Latency on the decoding (receiver) side

Vsync

Currently, video decoding on the receiver side is done use a modified hello_video.bin. hello_video.bin is an example program supplied by the Raspberry foundation for decoding h264 video using the Pi h264 hardware decoder.

The modifications to the original hello_video.bin (thanks a lot to RCGroups forum member "mmormota") make sure that the video playback is smooth by looking for frame delimiters in the videostream and delaying the playback of each frame until a new vsync has occured.

There are two issues with that though:

Looking for the frame delimiter is done "in software" using the main CPU, this adds significant processing overhead, i.e. the hello_video.bin process has higher CPU usage
Since the delivery of the frame to the GPU for decoding is delayed in userspace, this means, that the decoding of the frame takes place after the "correct" (i.e. when vsync just occured) time. This isn't a problem as long as the decoding time is constant, however, since the decoding time is not constant, it creates additional jitter and delay.

What could be improved about that:

Use of the GPU for parsing the h264 stream and looking for delimiters, see the links about the gst-mmal project below for more information and code examples.
Make sure that the frames get delivered to the GPU for decoding as soon as they are received (so that the GPU can already start decoding while we're waiting for the next vsync to occur).
Slightly change the HDMI clock when appropriate to compensate for the camera framerate clock on the air Pi and the refresh-rate clock on the ground Pi drifting apart.

Frame buffering

Several frames are buffered inside the GPU when decoding. According to different postings on the Raspberry forum, this can be reduced to one frame by setting some parameter (max_dec_frame_buffering=1). If we can reduce the buffering by only one frame, we would already save about 16ms of latency, two frames would be 32ms!