Recover the wifi instead of waiting for someone to notice, and make the daily video cheaper

CHANGELOG.md

@@ -8,6 +8,92 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Fixed
+- **Cameras that "froze" every week or two were losing their network, not
+  hanging.** Diagnosed on a Pi that had run 9.3 days: capture never missed a
+  frame — exactly 2880 a day, one gap over 60 seconds in ten days, CPU
+  temperature flat at 42-46°C and load flat at 0.34-0.38 throughout, no OOM
+  anywhere. The daily ffmpeg job, the obvious suspect, succeeded every single
+  day and finished two hours before the reboot.
+
+  What actually happened: the access point dropped at 23:31, both BSSIDs timed
+  out within 23 seconds, and NetworkManager read the timeout as a wrong
+  password — `need-auth` → `no secrets: No agents were available` → `failed
+  (reason 'no-secrets')`. That sets an autoconnect *blocked reason* on the
+  profile rather than a retry counter, so `connection.autoconnect-retries=-1`,
+  which was already set, could not clear it. wlan0 sat `inactive` for 8h35m
+  until a human power-cycled the Pi. The only symptoms were the upload and the
+  live image going quiet, which is indistinguishable from a hang unless you
+  look at the card.
+
+  `scripts/check_network.sh` now recovers it, opt-in via
+  `./scripts/install.sh --with-netwatch`. Two checks of grace, then `nmcli dev
+  connect` (the step that clears the block), the priority profile explicitly, a
+  radio cycle, a NetworkManager restart, and only then a reboot — gated on 30
+  minutes of continuous failure, the SSID having been seen on the air, and 6
+  hours since the last watchdog reboot, so an access point that is switched off
+  can never trigger one. `docs/TROUBLESHOOTING.md` gained "The camera looks
+  frozen but is still capturing".
+
+- **The capture watchdog was spamming the journal it would need later.**
+  `find | sort -rn | head -1` left `sort` writing into a closed pipe every run;
+  invisible interactively, but systemd sets `IgnoreSIGPIPE=yes`, so under the
+  service it printed two error lines every five minutes into a journal already
+  at its 200 MB cap. Diagnosing the outage above, `journalctl --list-boots`
+  only reached back two days of a nine-day run. Now one `awk` pass.
+
+- **`video.codec.name: h264_v4l2m2m` silently converted the 05:00 job into a
+  nightly failure.** The Pi's V4L2 encoder stops at 1080p and nothing sets an
+  output resolution by default, so it reached ffmpeg and died with "can't
+  configure encoder". Refused up front with an actionable message instead.
+
+### Added
+- **Memory, disk, uptime, process RSS and network state in `captures`**
+  (schema v6). `SystemMonitor` had collected memory and disk every 30 seconds
+  since it was written; `store_capture` bound the CPU temperature and three
+  load averages and dropped the rest. Ruling out a memory leak during the
+  outage above meant arguing from temperature and load alone. `system_uptime_s`
+  makes every reboot visible in the capture timeline itself, and
+  `network_signal_dbm` is the one that would have predicted the drop rather
+  than merely recording it. Migrating this camera's real 549,288-row database
+  took 111 ms.
+
+- **`video.retention_days`, defaulting to 7.** `cleanup_old_images.sh` has
+  expired source frames for a long time; nothing ever expired the videos made
+  from them, so a bounded input fed an unbounded output — 3.2 GB from twelve
+  days here. Runs as a third step of the existing nightly cleanup timer. Never
+  deletes a file the upload queue still holds in any state other than
+  `success`, including `failed`, because that video exists nowhere else.
+
+- **A shared reboot floor between the two watchdogs.** Both can reboot the
+  machine and a Pi with wedged wifi looks stalled to the other one, so
+  `/var/lib/raspilapse/last_reboot` now stops them cycling. Written with
+  `sync`, because ext4's commit window will otherwise lose it across the very
+  reboot it bounds.
+
+### Changed
+- **The daily video is both faster and closer to the source**: preset `fast` →
+  `veryfast`, crf 25 → 20, threads 2 → 3, still native 4K. Measured on 60 real
+  frames: 25.2 → 43.8 Mbit/s, 75s → 35s, SSIM 0.985634 → 0.990225. A preset
+  does not set quality, crf does, and at 4K the encode is dominated by decode
+  and filtering rather than bitrate — so the old settings were paying for a
+  slower preset and spending the savings nowhere. crf 25 at 4K quantises away
+  the fine texture 4K exists to carry. Peak memory *fell*, 1399 MB → 1021 MB.
+
+- **The keogram and slitscan share one decode pass.** Both are a vertical strip
+  from every frame; they were built by two separate full passes over the day's
+  4K images, 314s + 271s. `create_time_slices` does one, 2.20x faster on 300
+  real frames, with both outputs sha256-identical to what the split version
+  produced. `create_keogram` and `create_slitscan` remain as wrappers.
+
+- `CPUWeight=20` on the daily video unit, since `Nice` alone does not hold back
+  a multi-threaded encode. `MemoryHigh`/`MemoryMax` are set too, but note that
+  Raspberry Pi OS ships the memory cgroup controller disabled — they need
+  `cgroup_enable=memory cgroup_memory=1` in `cmdline.txt` to do anything.
+
+- A `flock` on `data/daily.lock` so a manual `raspilapse-daily` cannot run two
+  ffmpegs alongside the timer's. Contention exits 0, not 1: a duplicate
+  invocation is not a failure.
+
 - **A one-frame colour step at every dusk, and daylight colour that drifted a
   little each day.** Both came from AWB readings reaching the manual gains by
   paths that stopped making sense when white balance became manual on every

docs/CONFIG-REFERENCE.yml

@@ -131,19 +131,28 @@ video:
     # Pixel format (yuv420p for maximum compatibility)
     pixel_format: "yuv420p"
 
-    # Preset for libx264 (affects speed vs quality vs memory)
-    # ultrafast = fastest, lowest memory, acceptable quality
-    # fast = good balance (recommended)
-    # slow = best quality, highest memory (may OOM on 4K)
-    preset: "fast"
+    # Preset for libx264 (speed vs bits needed for a given quality)
+    # A preset does not set quality -- crf does. A faster preset reaches the
+    # same crf with a bigger file, which on a Pi encoding 4K is the trade you
+    # want: at 3840x2160 the cost is dominated by decode, scale and deflicker
+    # rather than by bitrate, so a slower preset buys very little.
+    # Measured on 60 real 4K frames from this camera:
+    #   fast     crf 25 -> 25.2 Mbit/s, 75 s   (the old setting)
+    #   veryfast crf 20 -> 43.8 Mbit/s, 35 s   SSIM 0.9902 vs 0.9856
+    # i.e. faster AND closer to the source. Disk is what pays, and with
+    # video.retention_days set that is bounded.
+    preset: "veryfast"
 
     # Thread count (lower = less memory, slower encoding)
-    # 2 = safe for Pi with 4GB RAM doing 4K
-    threads: 2
+    # 3 on a 4-core Pi leaves a core for the capture loop; 4 measured no faster
+    # here because the job is decode-bound.
+    threads: 3
 
     # Constant Rate Factor (0-51, lower = better quality)
-    # 18 = visually lossless, 23 = good quality, 25 = good balance, 28 = acceptable
-    crf: 25
+    # 18 = visually lossless, 20 = keeps 4K detail, 23 = good, 28 = acceptable
+    # At 4K, crf 25 quantises away the fine texture 4K exists to carry: the
+    # frame is still 3840x2160 but foliage, water and distant edges flatten.
+    crf: 20
 
   # Frame rate (frames per second)
   # 25 fps = smooth European standard
@@ -157,6 +166,20 @@ video:
   # Number of frames to average for deflicker (higher = smoother but may blur fast changes)
   deflicker_size: 10
 
+  # Delete generated videos, keograms and slitscans older than this many days.
+  # 0 keeps everything forever, which is what happened before this key existed:
+  # cleanup_old_images.sh expires the source JPEGs, and nothing expired what is
+  # made from them, so the video directory grew without bound.
+  #
+  # A file is never deleted while the upload queue still holds a row for it in
+  # any state other than 'success' -- including 'failed', because a video that
+  # nobody managed to upload exists nowhere else. Those are logged as retained
+  # rather than silently skipped. Files the queue has never heard of are not
+  # protected, which is how ad-hoc partial-day renders get cleaned up.
+  #
+  # At 4K/crf 20 a day is roughly 500 MB, so 7 days is ~3.4 GB.
+  retention_days: 7
+
   # Default time range for timelapse generation (used when no --start/--end provided)
   # Format: HH:MM
   # If end time <= start time, assumes start is from previous day

docs/TROUBLESHOOTING.md

@@ -163,6 +163,19 @@ default, in `scripts/cleanup_old_images.sh`). Database rows live longer —
 lux, brightness and weather history the graphs are drawn from. A row whose
 `image_path` no longer exists is expected, not broken.
 
+The **videos** made from those images have their own window,
+`video.retention_days`. Until that key existed the source frames were bounded
+and the thing they turn into was not, so the video directory grew forever — 3.2
+GB from twelve days on this camera. At 4K/crf 20 a day is roughly 500 MB, so 7
+days settles near 3.4 GB. A file is never deleted while the upload queue still
+holds a row for it in any state other than `success`, including `failed`;
+those are logged as retained rather than silently skipped. Check before
+trusting it:
+
+```bash
+python3 -m raspilapse.cli.prune_videos --dry-run
+```
+
 Reclaiming space after a large prune needs an explicit vacuum, which is slow and
 needs free disk equal to the database size:
 
@@ -264,6 +277,64 @@ lives in `/var/lib/raspilapse/` so it survives a reboot.
 
 ---
 
+## The camera looks frozen but is still capturing
+
+Check this **before** assuming the Pi has hung. The symptoms of a dead network
+and a dead Pi are identical from a desk: no new video uploaded, no live image on
+the server, nothing responding. But capture writes to the local card and does
+not care whether the network exists, so the camera can be perfectly healthy and
+completely invisible at the same time.
+
+Ask the card, not the dashboard:
+
+```bash
+ls -t /var/www/html/images/$(date +%Y/%m/%d)/*.jpg | head -3
+sqlite3 data/timelapse.db \
+  "SELECT datetime(unix_timestamp,'unixepoch','localtime') FROM captures
+   ORDER BY unix_timestamp DESC LIMIT 3;"
+```
+
+Recent timestamps mean the camera never stopped and the problem is the network.
+
+This is not hypothetical. On 6 August 2026 this camera's access point dropped at
+23:31, both BSSIDs timed out within 23 seconds, and NetworkManager read the
+timeout as a wrong password:
+
+```text
+NetworkManager: Activation: (wifi) disconnected during association, asking for new key
+NetworkManager: state change: need-auth -> failed (reason 'no-secrets')
+```
+
+`failed (reason 'no-secrets')` sets an autoconnect **blocked reason** on the
+profile rather than a retry counter, so `connection.autoconnect-retries=-1` does
+not clear it and neither does anything else in `NetworkManager.conf`. Only an
+agent supplying secrets, an explicit `nmcli con up` / `nmcli dev connect`, a
+change to the profile's secrets, or restarting NetworkManager will. wlan0 sat
+`inactive` for 8h35m until someone power-cycled the Pi — which "fixed" it and
+made a network fault look like a hang. Capture ran the whole time: 2880 frames
+that day, none missed.
+
+To recover automatically instead:
+
+```bash
+./scripts/install.sh --with-netwatch
+```
+
+Runs as root every 2 minutes. Two checks of grace so an access point reboot
+rides out untouched, then `nmcli dev connect`, the priority profile explicitly,
+a radio cycle, a NetworkManager restart, and only then a reboot. The reboot
+needs 30 minutes of continuous failure, your SSID seen on the air at least once
+during it, and 6 hours since the last watchdog reboot — so an access point that
+is genuinely switched off never triggers one, however long it stays off.
+
+Watch it decide without letting it act:
+
+```bash
+sudo env RASPILAPSE_NETWORK_STATE=/tmp/netstate bash scripts/check_network.sh --dry-run
+```
+
+---
+
 ## Tests fail after pulling
 
 ```bash

pyproject.toml

@@ -76,6 +76,7 @@ raspilapse-daily = "raspilapse.cli.daily:main"
 raspilapse-overlay = "raspilapse.cli.apply_overlay:main"
 raspilapse-db = "raspilapse.cli.db:main"
 raspilapse-retry-uploads = "raspilapse.cli.retry_uploads:main"
+raspilapse-prune-videos = "raspilapse.cli.prune_videos:main"
 
 [tool.setuptools.packages.find]
 include = ["raspilapse*"]

raspilapse/cli/prune_videos.py

@@ -0,0 +1,8 @@
+"""`raspilapse-prune-videos` -- delete rendered videos past the retention window."""
+
+from raspilapse.video.retention import main
+
+__all__ = ["main"]
+
+if __name__ == "__main__":
+    raise SystemExit(main())