Absolute timestamps

The timestamps you usually see when dealing with frames in GStreamer are relative to when you started recording or generating data, i.e. they're offsets. I wanted a way to later recover the real-world time at which a frame was captured.

I created a new pipeline element called absolutetimestamps and you can install it like so:

$ git clone [email protected]:george-hawkins/gst-absolutetimestamps.git
$ cd gst-absolutetimestamps
$ ./autogen.sh
$ ./configure --prefix=$HOME
$ make install
$ export GST_PLUGIN_PATH=$HOME/.gstreamer-1.0/plugins
$ gst-inspect-1.0 absolutetimestamps

To test it out try:

$ gst-launch-1.0 videotestsrc num-buffers=120 ! 'video/x-raw,width=1024,height=768,framerate=24/1' ! clockoverlay ! absolutetimestamps ! jpegenc ! avimux ! filesink sync=true location=out.avi

This pipeline creates 5 seconds of video with the test source videotestsrc and each frame has the current time overlaid onto it by the clockoverlay element. The resulting video is saved to out.avi.

At the same time the absolutetimestamps element will have generated a file called timestamps.log containing output like this:

0:00:00.000000000 2019-08-04T14:59:13.924907Z
0:00:00.041666666 2019-08-04T14:59:13.942342Z
0:00:00.083333333 2019-08-04T14:59:13.985726Z
...

Each line is a frame timestamp and the real-world time at which the absolutetimestamps element saw it.

If you want it to save this data to a different file you can specify the file location with the location property:

$ gst-launch-1.0 ... ! absolutetimestamps location=my-filename ! ...

Notes

Without sync=true at the end of the videotestsrc pipeline above, the resulting video plays back far far slower than real-time and the expected relationship between framerate=24/1 and num-buffers=120 doesn't hold, i.e. that we get 120 / 24 seconds of video. Without sync=true, it seems videotestsrc blasts out the specified number of buffers as fast as possible (in about 0.5s on my machine), but the rest of the pipeline is told that they were generated at 24 fps. The resulting file will contain frames captured over 0.5 seconds of time but play it back over 5 seconds. This means you probably won't see the seconds, generated by clockoverlay, tick during the replay as the original frames cover less than a second of time.

TODO: find out why sync=true at the end of the pipeline is needed to control the behavior of videotestsrc at the start of the pipeline and cause framerate=24/1 to have the effect one would have expected in the first place.

How this plugin was created

Getting started with GStreamer development has a high initial learning curve. This was my first GStreamer element and although in the end the work required was minimal, it was hard to get going. The GStreamer website does have a plugin writer's guide but I didn't find it particularly useful. After you work your way through the initial steps you reach this comment:

FIXME: this section is slightly outdated. gst-template is still useful as an example for a minimal plugin build system skeleton. However, for creating elements the tool gst-element-maker from gst-plugins-bad is recommended these days.

Perhaps it would have been more helpful to put this notice at the start. So the initial section and much of the rest of the information seems to be fairly stale.

For the very basics of getting started RidgeRun have more up-to-date instructions on their wiki. RidgeRun have lots of useful information about GStreamer and working with cameras on embedded systems and the like, so I'd tend to trust information they provide.

The following is a walkthru of creating this simple element based on the RidgeRun instructions and on looking at the source of plugins that did similar things to what I wanted, e.g. gstidentity.c and gstfilesink.c (the source for the identity and filesink elements respectively).

First, determine the version of GStreamer that you're system is using:

$ gst-launch-1.0 --version
gst-launch-1.0 version 1.14.4
GStreamer 1.14.4
https://launchpad.net/distros/ubuntu/+source/gstreamer1.0

Now clone the gst-plugins-bad repo (despite "bad" in the name, it is the GStreamer repo that provides the tooling for creating new elements) and checkout the appropriate version:

$ git clone https://github.com/GStreamer/gst-plugins-bad.git
$ cd gst-plugins-bad
$ git tag
$ git checkout 1.14.4

Initially the common subdirectory is empty. We need some of the scripts that autogen.sh creates their, normally autogen.sh also runs configure but we don't need this step:

$ ls common
$ NOCONFIGURE=true ./autogen.sh
$ ls common

You may find that some tools, e.g. autopoint, need to be installed with sudo apt install before autogen.sh can run through successfully.

Now add common to PATH. The tool gst-project-maker, that's used later, depends on the gst-indent script found there:

$ PATH=$PWD/common:$PATH

And gst-indent depends on indent, which may not be installed. Check and resolve this like so:

$ gst-indent --help
GStreamer git pre-commit hook:
Did not find GNU indent, please install it before continuing.
$ sudo apt install indent
$ gst-indent --help
usage: indent ...

OK - now we're ready to create the skeleton for our new GStreamer project using the gst-project-maker tool. It depends on the libgstreamer-plugins-bad1.0-dev, so let's install that and then run gst-project-maker to create a new project called absolutetimestamps:

$ sudo apt install libgstreamer-plugins-bad1.0-dev
$ cd tools
$ ./gst-project-maker absolutetimestamps
$ git status

The call to git status will show that a new subdirectory called gst-absolutetimestamps has been created.

Now we've got our overall project skeleton, let's create the boilerplate for our new element (with basetransform as its superclass, i.e. the same superclass that identity uses):

$ ./gst-element-maker absolutetimestamps basetransform
Plugin Details:
  Name                     absolutetimestamps
  Description              FIXME plugin description
  ...
$ git status

This time git status shows that file gstabsolutetimestamps.h and gstabsolutetimestamps.h (along with some other files) have been created.

Notes:

• The name you use, i.e. absolutetimestamps here, should be a valid C identifier, e.g. absolute-timestamps would result in invalid code being generated later by gst-element-maker.
• Here we've created a project called absolutetimestamps and an element called the same thing - but one could create a project with several plugins and tools, e.g. a muxer and demuxer or a sink and source.

Above we saw that gst-element-maker output some details about the plugin. If you look at gst-element-maker you'll see that at the end it builds the generated source into a .so and then runs gst-inspect-1.0 on this. It's gst-inspect-1.0 that generates the plugin details that you see.

We need to move the .h and .c file generated by gst-element-maker into our skeleton project and, as we're only building a single element and no tools, we need to clear out some of the skeleton code:

$ cd gst-absolutetimestamps
$ mv ../gstabsolutetimestamps.[ch] plugins
$ rm plugins/gstabsolutetimestampsplugin.c
$ rm -r tools

Note: you can create a project which contain several elements - in such a case you'd keep gstabsolutetimestampsplugin.c and register each of the elements there.

Now that we've removed gstabsolutetimestampsplugin.c we need to remove it from the _SOURCES list in plugins/Makefile.am:

$ vi plugins/Makefile.am

Similarly, we need to remove tools from SUBDIRS in Makefile.am and remove tools/Makefile from AC_CONFIG_FILES in configure.ac:

$ vi Makefile.am configure.ac

Now we can move gst-absolutetimestamps out of the current repo:

$ cd ..
$ mv gst-absolutetimestamps ../..
$ cd ../../gst-absolutetimestamps

OK - we're ready to create a new repo. However, you may first want to remove the files AUTHORS, NEWS, README and ChangeLog if you don't plan on maintaining them. You should also update the license in COPYING, and at the start of the .h and .c files in the plugins subdirectory, to match your preferences. If you do remove AUTHORS etc. you'll also have to modify configure.ac slightly:

$ vi configure.ac

Change the line AM_INIT_AUTOMAKE([1.10]) to become AM_INIT_AUTOMAKE([1.10 foreign]), i.e. add in foreign, otherwise, it'll complain about AUTHORS etc. not being present.

Once you've made any such changes, set up the initial repo:

$ git init
$ git add .
$ git commit -m 'Initial import.'

Now let's run autogen.sh for our new project:

$ ./autogen.sh

Among other things, this will create a Makefile where the installation directory for plugins is specified as /usr/local/lib/gstreamer-1.0. If you'd rather not install globally you can regenerate things like so:

$ ./configure --prefix=$HOME

There's no one-step way of doing this as autogen.sh doesn't take arguments. With prefix set like this, plugins end up in ~/.gstreamer-1.0/plugins.

Assuming you set prefix as above you can now build and install the plugin without sudo:

$ make install

This will create a lot of artifacts that you probably want git to ignore - so clean up the output of git status to create a suitable .gitignore:

$ git status > .gitignore
$ vi .gitignore
$ git status
$ git add .gitignore
$ git commit -m 'Added .gitignore to ignore outputs of ./autogen.sh and make.'

OK - assuming everything was installed locally, we can inspect the new element like so:

$ export GST_PLUGIN_PATH=$HOME/.gstreamer-1.0/plugins
$ gst-inspect-1.0 absolutetimestamps

Customization

At this stage you have a basic skeleton and a lot of boilerplate - now it's time to customize this to achieve whatever you need.

First, replace the various FIXME and fixme reminders in the .h and .c files to something more appropriate. You can see the changes I made, for this step, in commit fd7c2c9 - I just copied the wording, capitalization style etc. that I saw in existing GStreamer elements.

I wanted an element that just passed data through unchanged so, as noted above, I looked at the source for gstidentity.c and by trial-and-error, I arrived at a minimal set of changes that behaved like the identity element, in that the plugin could be included in a pipeline and would just pass on its data.

These changes can be seen in commit b01f7a6 - they just involve knocking out some of the boilerplate methods that override the base class behavior and updating the caps handling.

Update: later I went back and removed all remaining stub functions, generated by gst-element-maker, that didn't need to be modified to provide functionality for this element - see commit 084af0b.

When testing my changes during development, I used a pipeline like this:

$ gst-launch-1.0 videotestsrc num-buffers=120 ! 'video/x-raw,width=1024,height=768,framerate=24/1' ! clockoverlay ! absolutetimestamps ! jpegenc ! avimux ! filesink sync=true location=out.avi

Then I added the new functionality that I wanted - for each frame I wanted to print out the real-world time and the timestamp of the frame. This minimal bit of functionality just needed to be added to the transform_ip method and can be seen in commit e0c8c3c.

Finally I wanted the mapping from real-world times to timestamps to be written to a file. I wanted to specify the file location as one does with the filesink element, so I used the approach used in gstfilesink.c as the basis for this code. The result can be seen in commit 8acf1fa.