System setup

Tips for setting up your Linux system running Moonfire NVR.

Realtime clock on Raspberry Pi

Most Linux systems have a built-in battery-backed "realtime clock" (RTC). Since Linux 2.6, the kernel sets the system time on startup from this clock. This means that the time seen by applications will always be approximately correct, even after a long power outage. Later in the boot process a time syncer application (eg systemd-timesyncd, chrony, or ntpd) fetches exact correct time over the network. Often, the correction will be mild and done by "slewing" (adjusting the clock rate) rather than "stepping" (changing the current time).

The Raspberry Pi doesn't come with a RTC. Instead, /etc/init.d/fake-hwclock saves the current time to a file (hourly via /etc/cron.hourly/fake-hwclock and on shutdown via /lib/systemd/system/fake-hwclock.service) and loads it back on startup (via /lib/systemd/system/fake-hwclock.service again). This is better than thinking the time is January 1st 1970 on startup but can produce confusing results with Moonfire NVR. After a two-hour power outage, the time may be three hours behind on startup. When the time syncer fetches the exact time, the system time will be stepped, but likely too late. Moonfire NVR will have already started recording streams and noted their starting timestamp from an incorrect clock. Streams will keep having an incorrect time until you manually restart Moonfire NVR.

To fix this, you can buy an add-on RTC for your Pi such as this DS3231 module. It connects to your Pi's I2C bus via the GPIO header. Setting it up can be a little tricky. There are various blog entries describing how to do it. As of 2021-07-13, I find this following approach works on the Raspberry Pi OS 64-bit beta (based on Debian 10 Buster):

Create /etc/addon-hwclock

#!/bin/bash
# Use add-on hardware RTC module. See /etc/systemd/system/addon-hwclock.service.

# Be verbose so failures can be examined with `journalctl --unit add-hwclock`.
set -o errexit
set -o xtrace

if [[ "$1" = "start" ]]; then
    # Load the kernel modules needed by the RTC.
    # udevd might do some/all of this later, but we want it now (before root fsck).
    for i in i2c_bcm2835 rtc_ds1307; do /sbin/modprobe $i; done

    # Potentially useful debugging commands. Uncomment to taste.
    # /usr/bin/dtc --in-format fs /proc/device-tree
    # /usr/sbin/i2cdetect -y 1

    echo ds1307 0x68 > /sys/class/i2c-adapter/i2c-1/new_device

    # Debugging, again.
    # /usr/bin/dtc --in-format fs /proc/device-tree
    # /usr/sbin/i2cdetect -y 1

    /sbin/hwclock --hctosys --utc --verbose

elif [[ "$1" = "stop" ]]; then
    /sbin/hwclock --systohc --utc --verbose
fi

Create /etc/systemd/system/addon-hwclock.service

[Unit]
Description=Add-on hardware RTC module
DefaultDependencies=no

# Run after fake-hwclock so it doesn't override the time set from the real hwclock.
After=fake-hwclock.service

# Filesystems record their last mount time; fsck complains if it's in the future.
# Run before the first fsck to avoid this. Earlier in the boot process is better anyway.
Before=sysinit.target systemd-fsck-root.service
Conflicts=shutdown.target

# Note because of the WantedBy=sysinit.target, the system will boot into
# emergency mode on failure. Be a little defensive with the ConditionFileIsExecutable
# to avoid that annoying failure mode.
ConditionFileIsExecutable=/etc/addon-hwclock

[Service]
Type=oneshot
RemainAfterExit=yes

ExecStart=/etc/addon-hwclock start
ExecStop=/etc/addon-hwclock stop

[Install]
WantedBy=sysinit.target

Activate them

sudo chmod a+rx /etc/addon-hwclock
sudo systemctl enable addon-hwclock.service

That covers the stuff that should happen on each boot. Before rebooting, you'll also want to run a variation of those commands to set the hardware RTC properly the first time:

sudo sh -c 'for i in i2c_bcm2835 rtc_ds1307; do /sbin/modprobe $i; done'
sudo sh -c 'echo ds1307 0x68 > /sys/class/i2c-adapter/i2c-1/new_device'
sudo /sbin/hwclock --systohc --utc --verbose

Then reboot and check sudo journalctl -b0 to ensure it works. You should see a line that says Time read from Hardware Clock before the line that says Starting File System Check on Root Device.

Why not other ways?

• This thepithut.com guide suggests setting the RTC from /etc/rc.local. But that happens later in the boot process, as defined by /lib/systemd/system/rc-local.service. We want the time to be set correctly before the root filesystem's fsck, and certainly before Moonfire NVR starts.
• This learn.adafruit.com guide suggests:
  • setting dtoverlay=i2c-rtc,ds1307 or dtoverlay=i2c-rtc,ds3231 in /boot/config.txt rather than running an echo ds1307 0x68 > /sys/class/i2c-adapter/i2c-1/new_device command. This approach had no apparent effect on my system. I don't know why. There's raspberrypi.org devicetree documentation and kernel.org devicetree documentation of this mechanism; I haven't figured out what went wrong. I see a related bug which should be long-fixed. In any case, the new_device way worked better for me.
  • depending on /lib/udev/hwclock-set. It's started by /lib/systemd/system/system-udevd.service (which eventually launches it via /lib/udevd/rules.d/85-hwclock.rules). systemd-udevd.service doesn't start until after the initial fsck, and as far as I can tell nothing waits for its rules to complete before completing system startup.
  • modifying /lib/udev/hwclock-set. This is file is part of a system package, so upgrades may clobber your local changes.
  • disabling fake-hwclock.service: it's better than nothing, so I leave it in case something goes wrong with the hardware module setup. But the downside leaving fake-hwclock.service in place is that journalctl log timestamps are wrong for the first several lines of the boot, so you might instead run systemctl disable fake-hwclock.service and remove the After=fake-hwclock.service from /etc/systemd/system/addon-hwclock.service.
• This askubuntu.com answer sets up a systemd service:
  • It omits the Before=systemd-fsck-root.service so doesn't affect time during the initial fsck.
  • It races with fake-hwclock.service so its time may be overridden.
  • It depends on systemd-modules-load.service (and thus probably expects a /etc/modules-load.d/ file listing the relevant modules).
  • It creates a file in /lib/systemd when man systemd.unit says locally-installed files should go in /etc/systemd instead.
  • I'm unsure how /dev/rtc0 is created on there; I don't see anything which creates a device tree mapping.