Capture prior OS state before tuning, and restore it exactly on revert

[rammsguns]

Closes #23.

The rollback now restores the state this machine was actually in, rather than a
set of defaults someone assumed. New library lib/ostune.sh; 10-os-tune.sh
and 19-os-revert.sh rewritten around it.

What was wrong

19-os-revert.sh wrote fixed values — THP=madvise, governor=schedutil,
power at maximum, persistence off — and rm -f'd three paths under /etc it
had never proven it owned. On a machine with a governor policy, a THP setting,
or its own 99-llm-inference.conf, that was not a rollback but a second round
of configuration, and in the file case a deletion.

The one worth naming: restoring the maximum power limit raises a cap someone
deliberately lowered. TUNING.md argues 100% is right for this chassis; a
reader who disagreed and capped their card at 120W had that undone by a script
advertised as an undo.

Acceptance criteria

Criterion	Where
Capture effective prior state before the first mutation	ostune_state_put immediately precedes every change in 10-os-tune.sh
Never overwrite or delete a pre-existing unowned file	ostune_file_status / ostune_install_file — backed up byte for byte, or refused
Ownership/state marker, root-owned, restrictive permissions	/var/lib/llm-rig/os-tune.state, 0600, in a 0700 directory
Revert restores captured values, and is idempotent	19-os-revert.sh reads only the state file; a second run reports "nothing to revert" and exits 0
Partial failure still leaves a safe rollback	Capture is per setting and precedes its mutation; a test makes step 5 refuse and then reverts steps 1–4
--dry-run / plan mode, no sudo	Both scripts
Fixture tests for every listed case	tests/cases/ostune_test.sh, 35 tests
Docs match the implementation	README § What "reversible" means, TUNING.md § Rolling the OS tuning back

Design decisions worth disagreeing with

Capture is append-once. A second tune must not re-record llm-rig's own
values as the ones to restore — that is exactly how a rollback becomes a
silent no-op. The cost is that a genuine change of prior state between two
tunes is not noticed; the first capture wins.

Adopt-and-restore, not refuse, for pre-existing files. The issue allowed
either. Refusing would block the tune on any machine with its own sysctl file,
which is common enough that people would work around it by deleting theirs.
Backing up byte for byte and restoring byte for byte keeps the tune usable and
the guarantee intact. Fail-closed still applies: if the backup cannot be made,
nothing is written.

Unproven ownership blocks deletion. Ownership is recorded before the write,
with the hash filled in after. A crash in between leaves unknown, and revert
then refuses to delete. Safe direction, mildly annoying outcome: an orphaned
file the user must remove by hand, with a message saying so.

Per-CPU governors are captured and restored individually, because a machine
can legitimately run different governors on different cores and flattening them
to one value is a change, not a restoration.

An unreadable prior value is unknown and is never "restored". Writing
madvise because THP could not be read is a new decision.

A real bug the tests found

Existence was being inferred from hashability, so a path that exists but cannot
be hashed — unreadable file, directory, dangling symlink — read as absent and
was overwritten. That is the moment we know least about a path. It now refuses.
-L as well as -e, since -e follows symlinks and is false for a dangling
one: precisely the case where a write replaces a link with a regular file.

Tests

35 new, 412 across 13 suites, ordinary and network-isolated both green.

OSTUNE_ROOT points at a sandbox and OSTUNE_SUDO is empty, so the scripts run
their real code paths — writing, hashing, backing up, restoring, refusing —
against a fake root. Deliberately not mocked: a mock that always succeeds cannot
show that a file came back byte for byte.

Two testing notes:

• The cpupower mock now actually writes the sandbox governor files. A stub
  that only records the call cannot demonstrate that a revert put the previous
  governor back.
• The backup-failure case uses a dangling symlink rather than a chmod 000
  file, because tests/isolated.sh runs inside unshare --map-root-user where
  permission bits do not apply and an unreadable file is readable. It also uses
  the real ln rather than the mock on PATH, which would have created nothing
  and left the assertion measuring the stub.

Not done

No run against a real /etc or real GPU is included — this host has not had
10-os-tune.sh applied, and doing so to produce a transcript would change
machine state to demonstrate a change of machine state. The sandbox exercises
every branch; the first privileged run is still the first privileged run, and
--dry-run exists so it can be inspected before it happens.

🤖 Generated with Claude Code