w698: every critical unit in the portal VM starts after a reboot; fff-health restarts any that is down

[bryding]

Request: w698

TL;DR: fff-ops.socket did not come back after the VM's cold restart because it was ordered after a service, which is an ordering cycle at boot: systemd deletes a job to break it and the socket never starts. A deploy (install.sh restarts it by hand) hid this until a reboot. This PR removes the cycle, makes every critical unit enabled and checked at install, adds a unit watchdog to fff-health that restarts any failed or stopped critical unit (with back-off, a log line and a record that fffctl status and the portal show), and fixes the two warnings lothsahn saw (not joined, layer 7). Needs a portal deploy (fffctl update).

Root cause

• deploy/vm/guest/units/fff-ops.socket had After= and Requires=fff-ops-scratch.service. A socket unit is Before=sockets.target, sockets.target is before basic.target, and a normal service is After=basic.target: basic.target > sockets.target > fff-ops.socket > fff-ops-scratch.service > basic.target.
• Measured (reproduced offline, systemd 255, the repo's own units, SYSTEMD_UNIT_PATH=<units> systemd-analyze verify default.target): basic.target: Found ordering cycle on sockets.target/start, ... Found dependency on fff-ops.socket/start, ... Found dependency on fff-ops-scratch.service/start, ... Job sockets.target/start deleted to break ordering cycle starting with basic.target/start. The fixed unit prints nothing. deploy/vm/test/fff-units.test.sh runs this check on the boot graph and also proves it fails the old socket.
• install.sh ends with systemctl restart fff-ops.socket whenever it is not active, and a restart after boot has no cycle to break: each fffctl update revived the door, each reboot killed it.
• The Requires= also stopped the socket whenever fff-ops-scratch.service was stopped or restarted.
• Not established: the VM's own journal for 2026-10-08 (not readable from here), so the 15:33Z time in the report is not explained by this code alone; the commands that would show it are in the w698 report.

What changes

• fff-ops.socket: no ordering on any service (the per-connection fff-ops@.service already requires the scratch).
• FFF_CRITICAL_UNITS (lib.sh): firewall, tailscaled, scratch, socket, portal, update path, health/backup/base-refresh timers. install.sh enables the whole list and dies if one is not enabled after. tailscaled is enabled explicitly (it relied on the package postinst).
• Restart=on-failure + StartLimit* on fff-guest-firewall and fff-ops-scratch; tailscaled drop-in (Restart=on-failure, 5 s, 10 starts / 10 min). fff-ops-scratch, fff-ops@ and fff-backup.timer were heredocs in install.sh; now units/*.in templates (render_unit), so they are linted.
• fff-watchdog (run by fff-health every 30 s): a critical unit that is failed, inactive when it should run, or active without doing its job (no nftables table, not mounted, no socket file) is restarted; a disabled one is enabled. Back-off 30 s doubling to 900 s, stops after 6 restarts that did not hold (dispatcher told once), starts over after 600 s healthy. Skips: VM shutting down, unit starting/stopping or with a job, fffctl watchdog pause, the held portal. Every restart: journal tag fff-watchdog and data/unit-watchdog.json.
• fffctl status prints the unit table and the last restarts (unit, when, why); fffctl units [--check], fffctl watchdog run|pause|resume|reset. The portal shows a banner for a unit it gave up on, a dismissible one for the last 24 h of restarts, and system_status lines (server/unitWatchdog.ts, web/src/App.tsx).
• fff-vm watch (host): new layer units (fffctl units --check), the check on the watchdog from outside (it cannot restart its own timer). Host install.sh verifies its units and the VM's autostart are enabled.
• fffctl status "not joined": was the || echo 'not joined' fall-back of tailscale status --self | head -n 1, which fires whenever that pipe fails under pipefail, so a joined node printed its address and then not joined. Now reads Tailscale's BackendState. (Which of the two failure causes hit lothsahn's run needs the VM; the old code cannot say it means "not joined".)
• fff-vm watch layer 7: counts only dropped TCP from tag:ingress peers (Funnel is TCP; ICMP/UDP drops are shown, never judged), and is a warning with the ports while the outside probe passes. Message of the failing case unchanged.

Evidence

• deploy/vm/test/fff-watchdog.test.sh (new, 11 scenarios, fake systemctl): all pass in Ubuntu 24.04 (WSL).
• deploy/vm/test/fff-units.test.sh (new): passes; fails with the old socket (checked).
• fff-vm-path.test.sh 46 passed, fff-vm-nightly.test.sh 9, fff-machine-ssh.test.sh 7; shellcheck 0.10 -S warning clean on all touched scripts; npm run typecheck (root + web) clean; server/unitWatchdog.test.ts + pathHealth + secretGuard 46 pass.
• Not run locally: lint.sh --units (root, throwaway machine) and the nested-VM end to end (new step: after the nightly cold restart every unit is up, no ordering cycle in the boot journal, a stopped fff-ops.socket and a disabled fff-update.path are put right). CI runs both.

After the merge

Waiting for lothsahn's deploy (fffctl update; it can carry #234, w694). Verification steps are in the w698 report.

🤖 Generated with Claude Code