hostlock: fix a zombie-anchor wedge and a live-holder theft, and make the seven properties falsifiable

[justinchuby]

The advisory host lock is merged (73c76458c, then 6a4db3f24 from Gaff's review). This PR closes the conformance gap against the seven required properties — and, in doing so, found two live defects in the merged implementation.

Three retractions, in order

An earlier revision of this PR said "No production change. The merged implementation already satisfies all seven requirements." That was wrong, and it was wrong in the reassuring direction. I wrote it after reading the code rather than after trying to break it. Opus review rejected the claim and named two of the seven as unproven; I reproduced both, and both are real bugs that were on main.

And then the fix for one of them was itself worse than the bug. A second review round found that rejecting process state Z reports a live holder dead — see "Defect 3" below. The first bug wedged the box; my fix let one agent steal another's box mid-benchmark, which is strictly the worse error and the exact thing the tool exists to prevent.

And the fix for that had the same shape again. The anti-theft guard I added in round two returned before the holder's own TTL was consulted, so a lock with a live anchor and no readable start time outlived any ttl forever — a bounded wedge replaced by an unbounded one, while fixing a theft. See "Defect 6".

And it recurred a fourth time, in the fix for that. The class round three made newly reapable — a live anchor, no start_time, a lapsed ttl — was then announced as a dead holder: status said "is gone" about a pid the guard had just verified was running, and the reaper logged "reaping stale lock from dead pid N", which suppresses the "still alive … both sets of numbers are now suspect" warning in exactly the case it applies to. Same defect, third arm. See "Defect 7". The same round found that the guard's liveness test ([ -d /proc/$a ]) accepts a zombie, so on the run path (ttl=0) a corpse got the anti-theft protection and the box wedged forever — strictly worse than the theft the guard replaced. See "Defect 8".

I also have to correct the previous commit message: it claimed "EXPIRED would be unreachable for that whole class" was fixed by round three. It was not. EXPIRED requires holder_alive, which requires a start_time, so it stayed unreachable for that class; only the wait half of that claim was true. This commit is what actually makes it reachable.

This is the same over-claim pattern four times: the conclusion that says "your existing data is fine", the fix that says "and now it's handled", the guard that says "and nothing can take it now", and the release note that says "and this whole class is handled". Each was published without a falsification attempt, and each was caught by review rather than by me. Recording all four rather than quietly editing the body.

Defect 1 — R6 (SIGKILL recovery) was not satisfied: a zombie anchor wedges the box permanently

holder_alive decided liveness from two facts: /proc/<pid> exists, and its start time still matches. A SIGKILLed process whose parent has not wait()ed for it is a zombie — and a zombie keeps both. So the lock reports HELD on a corpse, forever.

Reproduced directly (before → after on the same script):

	status	acquire
before	HELD indefinitely	rc=2 BUSY, no escape
after	STALE	rc=0, takeover

It is worst on the run path, which sets ttl=0 reasoning that "its own pid is exact". Exact — and still resolving, to a dead process. With ttl=0 there is no expiry escape hatch either, so the box stays locked until someone deletes the directory by hand.

This is the common shape here, not the exotic one. Every agent harness on this host launches long commands via subprocess.Popen without an immediate wait(). Roy's prefill_ab.py is one.

The damning detail: hostlock_test.sh's own alive() helper has excluded state Z since the day it was written. The test file knew the correct semantics; the implementation did not. Fix reads state (field 3) and starttime (field 22) from /proc/<pid>/stat in one pass and rejects Z.

Defect 2 — R5: the anti-theft grace did not cover its own stated rationale

reapable has an UNPARSEABLE_GRACE window whose comment names the case it exists for: a lock written by an older or newer version of the script. But the guard tested [ ! -s "$META" ] — zero-byte only. A meta with content but no readable anchor_pid/start_time — exactly that case — skipped the grace, fell through to ! holder_alive, and had a live holder's box stolen out from under it.

Reproduced: a demonstrably live holder reported STALE, acquire returned rc=0 and took the lock. After the fix, rc=2 and the owner is preserved. Live risk right now, because we are running different revisions of this script side by side in different worktrees.

Also aligned lock_state: an unverifiable-but-ungraced lock now reports HELD, matching what acquire actually does. Still exactly four states.

Defect 3 — my own fix: state Z is not proof of death

When a thread-group leader exits via pthread_exit() while other threads keep running, /proc/<tgid>/stat reports Z for a fully live process. ps shows Zl ... <defunct>; Threads: says 2. Reproduced directly:

LIVE-but-Z leader:  pid=701267  state=Z  Threads=2   ps: Zl python3
TRUE zombie:        pid=681666  state=Z  Threads=1

So the previous commit's [ "$state" != Z ] || return 1 reported a live holder dead, and the next acquire reaped it mid-run. Reachable via acquire --pid <harness> — the flag R1 tells everyone to use. Threads: separates the two cases exactly; an unreadable status file now fails toward alive.

Defect 4 — the widened grace only deferred the theft it was meant to stop

reapable threw away a readable, running anchor_pid and fell back to lock age. The grace is 300 s; the runs it protects are 40 minutes. So a live holder running a different script revision kept its box for five minutes and lost it at 301 s — the fix covered 5 minutes of a 40-minute exposure, and my R5 test asserted at t≈0 only. It now returns "not reapable" whenever the anchor pid is present and running, and consults the clock only when there is no usable pid.

Related, same species as the retraction above: a takeover that happened only because a lock aged out of the grace now reports takeover=unverifiable, not stale_pid. The script never established the holder was dead.

Defect 5 — status told a blocked agent the box was abandoned

lock_state was fixed to say HELD for an unverifiable lock; cmd_status's human branch still keyed off holder_alive. cmd_acquire dumps that text on the BUSY path, so what a blocked agent actually read was:

hostlock: BUSY
STALE  holder alice pid=680653 is gone; next acquire will reap it

with pid 680653 alive. Telling a human the box is abandoned is how the box gets taken by hand. Driven from lock_state now.

Defect 6 — the anti-theft guard voided the holder's own TTL

Round two added: a readable, running anchor pid is never reaped on the clock. It returned before holder_expired was consulted. So a lock with a live anchor and no readable start_time outlived any ttl forever — wait blocked on a lock everyone agreed had expired, EXPIRED became unreachable for that whole class of lock, and the file's own contract (--ttl S hard expiry: a lock older than this is reapable by the next acquirer) was silently void.

The anti-theft property only needs the clock-based grace disabled, not the expiry the holder itself asked for. Verified with ttl=1, acquired 4000 s ago, live anchor: rc=2 before, rc=0 now.

Defect 7 — the third arm: a live holder announced as a dead one

Round three made the live-anchor/no-start_time/expired-ttl class reapable, and then described that takeover as a reap of a corpse:

$ hostlock.sh status                    # pid 3312 is RUNNING
STALE  holder alice pid=3312 is gone; next acquire will reap it
$ hostlock.sh acquire --owner leon
hostlock: reaping stale lock from dead pid 3312 (owner alice)

A takeover from a live holder is the one event this tool can cause that corrupts somebody's benchmark, and it is the one message that must never be downgraded to routine housekeeping. The correct output is the warning that already existed and was being skipped:

EXPIRED (held 4000s > ttl 1s; next acquire will take it over) by alice pid=3312 …
hostlock: WARNING taking over a lock held by alice (pid 3312, still alive) after its 1s TTL expired
hostlock: WARNING if alice is still benchmarking, both sets of numbers are now suspect

This is the third arm of one defect: HELD said "is gone" (fixed in commit 2), EXPIRED said "is gone" (fixed in commit 3), STALE said it in the reaper (fixed here). The root cause is that four call sites each decided "is this pid alive?" for themselves. Liveness is now one predicate, pid_is_live(), shared by holder_alive, the anti-theft guard, lock_state and the reaper, so they cannot disagree.

Defect 8 — the anti-theft guard protected a corpse, forever

The guard tested [ -d "/proc/${a}" ]. A zombie passes that: /proc/<pid> exists until the parent wait()s, and no parent here ever does. On the run path ttl=0, so holder_expired is never true — meaning the clock grace, the only thing that would ever have released the box, was disabled on behalf of a dead process. Unbounded wedge, in the guard written to prevent an unbounded wedge.

Fixed by routing that guard through the same pid_is_live() as everything else, so a dead anchor falls through to the clock. Verified with a real zombie (fork, SIGKILL, no wait) at ttl=0: BUSY inside the 300 s grace, recoverable past it.

Mutations that stayed green — my assertions were weaker than my claims

Two in the first round, five in the second, five more in the third, three more in the fourth. All found by review, none by me.

runnable_now() { echo 1; } — suite stayed green. My R2 test only pinned the occupancy figure into [1,10000] and never compared it to load the test itself created. The consequence is not a cosmetic column: a constant 1 silently satisfies every --gate N for N>=1, disabling the only mechanism that sees load from agents who never took the lock. A -f4→-f2 field slip is equally invisible. Now the test starts six bounded spinners and requires the reported occupancy to move with them.

pkill -9 -f ... inserted into reap_if_dead — suite stayed green. My R7 structural guard was grep -c '^[^#]*\bkill\b', which has three holes: \b does not match pkill/killall/killpg; -c counts lines, so a second kill appended to the sanctioned line is invisible; and ^[^#]* cannot cross a #, so a kill placed after any # anywhere on the line — including inside a string — is hidden. That last one fails in the false-PASS direction. Comments are now stripped first, then the whole kill family is counted by occurrence — except that the repair had the same false-PASS hole as the original: echo "reaping # " ; kill -9 "$p" is one shell command and no comment at all, but stripping at any # that begins a word truncates it. Only whole-line comments are stripped now, so a kill named in a trailing comment counts — a false FAIL, the safe direction — and timeout -s/timeout -k and xargs are matched. It is a net for concrete spellings, not a proof: K=kil; "${K}l" -9 $p evades any grep by construction, which is why the count is exact rather than a threshold and why the behavioural orphan test exists.

Five more green mutations, second round:

• "the wrapped command is really stopped" was inert. It ran after the runner had exited and been reaped, and an orphan reparents to pid 1, so pgrep -P "$runner" answers 0 whether or not the child was killed. Reproduced by making run_teardown never signal: test passed, sleep 60 still ALIVE-ORPHANED under pid 1. Now the child pid is captured before signalling.
• held_uid was compared against the test's own id -u. Everything here runs as one user, so uid=$(id -u) in the writer — reporting the reader's uid, destroying the field's only purpose — passed. Now anchored to pid 1, which is root and is never us.
• runnable_at_acquire only had to be numeric. echo "runnable_at_acquire=1" passed, in the very block whose comment argues at length that "is a number" is satisfied by a constant. The load-based version I first wrote could not distinguish stored at acquire from re-read at print time on a host busy for other reasons — and this host is. The stored value is now forged on disk and must be echoed verbatim.
• UNPARSEABLE_GRACE was entirely unpinned. Every dependent assertion created its lock immediately before checking, so now - mtime == 0 and any grace ≥ 1 passed; cutting 300 → 2 left the suite green. Now backdated on both sides of the window.
• run_teardown's start-time comparison had no test at all, so it could be weakened back to "the pid exists" silently.

Five more green mutations, third round:

• No fixture had an anchor_pid that was present and dead with no start_time. Every other one either carries a start_time or has no anchor_pid at all. So deleting [ -d "/proc/${a}" ] from the new guard — which makes every unparseable lock permanently unreapable, the exact wedge this branch exists to avoid — was invisible. The tell was already in the file: a dead_pid computed and then discarded with : "$dead".
• The teardown start-time check was pinned by grep -c, which counts lines. Appending || [ -d "/proc/$child" ] leaves the literal intact and the count at 1, restoring "the pid still exists" at the one place this script signals anything. Now asserted as a canonicalised golden function body — brittle to legitimate refactoring, which is a false FAIL and the safe direction.
• Only the HELD arm of cmd_status's human branch was asserted. EXPIRED is by definition a lock whose holder is alive, so putting "is gone; next acquire will reap it" there is the same false-abandonment message one arm over.
• UNPARSEABLE_GRACE was bracketed at 30 s and 4000 s — two orders of magnitude around 300. Now 200 s and 400 s.
• timeout --signal=, timeout --kill-after= and fuser -k evaded the R7 family regex. Unlike the acknowledged ${K}l indirection these are plausible future edits ("escalate the teardown"). Added.

Three more green mutations, fourth round:

• cmd_status's STALE arm had no assertion at all. Replacing its entire body with echo "FREE (nobody is here)" left the suite green — every STALE check in the file went through porcelain state=. That is the one arm where saying FREE is actively dangerous: a stale lock still covers a host that may be loaded, and "next acquire will reap it" is the sentence that sends a human through the tool instead of round it.
• The "present and dead" fixture was neither. dead_pid() is fully reaped, so /proc/<pid> does not exist — it is the absent-and-dead shape, and it silently tested a different branch than its comment claimed. The genuinely present-and-dead shape is a zombie, which is what Defect 8 turns on, and there was no fixture for it. Both shapes are now tested, and they reach the clock by different routes.
• cmd_run's trap ordering was unpinned. The golden function body added in round three is a body, not a position: moving RUN_CHILD_START=$(…) back above the three trap lines leaves that golden text byte-identical while reopening the window it exists to close — a signal arriving during the fork that reads the child's start time takes bash's default action, leaking the lock and orphaning a forty-minute benchmark. Now asserted positionally.

Two more, found by the new assertions rather than by review:

• <"/proc/$pid/stat" 2>/dev/null silences tr, not bash. Redirections apply left to right, so the input redirection fails and is reported before 2> is applied: every status on a stale lock printed two lines of No such file or directory in front of its output, and anything parsing that output got them too. Found because the new STALE text assertion is anchored — ^STALE against 2>&1. 2>/dev/null now comes first, and "status writes nothing on stderr for a departed pid" is asserted directly.
• The primary assertion for the primary fix could pass vacuously. "A live holder with no start_time still honours its own ttl" asserts acquire returns 0; if any earlier assertion in that block had already let the lock be reaped, it returns 0 for the wrong reason. It now asserts the fixture's starting state first.

And the coverage-count hole itself. Two checks sit behind environment probes (/proc/1 readability, uid), and an assertion that quietly stops running is indistinguishable from one that passes — the same failure mode as the inert R1 block and the unasserted STALE arm. Both probe branches now assert something, so the total is environment-invariant, and the total is pinned: deleting any single chk from the file now fails the suite by name. Verified by deleting one.

Also fixed, and not a mutation: the pid-1 held_uid check hardcoded 0 and asserted id -u != 0. That fails outright as root — the default in most CI containers — and wherever pid 1 is not root, for reasons that have nothing to do with this script. The expectation is derived from /proc/1/status now, and the check skips itself when pid 1 is unreadable or is us, because then it discriminates nothing.

The seven requirements

#	requirement	before	now
R1	holder is the outer harness, spanning all A/B/null arms	untested	asserted, and the rejected design pinned as rejected
R2	rows carry lock state, owner identity, occupancy snapshot	partial	occupancy must track load the test creates
R3	FREE / HELD / STALE / EXPIRED distinct	isolation only	distinctness asserted
R4	fail closed on wait and admission expiry	admission only	both, incl. acquire --wait (the path run uses)
R5	dead-holder PID + start-time reaping	one direction	both, plus a fixed theft bug, and takeovers from a live holder are warned about rather than logged as reaps
R6	SIGKILL recovery	forged lock, never a real kill	real kill and a real zombie, plus two fixed wedge bugs
R7	no process killing	untested	behavioural + structural, kill-family-wide

R1 — the old test was inert

The previous R1 block passed unchanged with its arms deleted and with $$ substituted for $BASHPID. Passing --pid explicitly makes a test insensitive to every anchoring decision it claims to be about. Rewritten as two harnesses differing only in which pid is the anchor, arms exiting normally, sampled in the gap between arms — which is exactly the moment Sebastian ran ps, saw no bench process, and concluded the host was clear. Per-arm anchoring must read STALE; harness anchoring must read HELD; and the two anchorings are asserted to genuinely disagree.

R6 — a forged lock never exercises the transition

Fabricating a lock from an already-dead PID asserts the reading of a stale lock, not the becoming of one. And my first attempt at a real kill still hid the zombie bug, because sig ...; wait ... reaps the zombie — it measured bash's reaping, not the lock's recovery. The test now keeps a real zombie alive (a parent that Popens, SIGKILLs, and never wait()s), confirms state Z, then requires HELD → STALE → rc=0 takeover with ttl=0.

R7 — the requirement whose violation would be catastrophic

Reclaiming a lock does not stop the load; the dead holder's benchmark is still on the cores. The tempting "fix" is for the reaper to kill it — which would make this tool capable of destroying a colleague's forty-minute run on the strength of a misparsed PID. A reap must leave the orphan running. The orphan is now named via exec -a, so a pattern-kill is caught behaviourally as well as structurally.

Also in this PR

• provenance emits runnable_at_acquire. The data was already written by publish_lock and silently dropped on the way out, so a row could only sample occupancy after the measured window closed. It can now bracket it.
• provenance emits held_uid, read from /proc/<anchor>/status. This is the honest half of R2's "owner identity": --owner is self-declared free text and corroborates nothing — --owner roy from my shell produces a row that says roy. The uid is the one identity the kernel vouches for. This is an advisory lock, so identity is for attribution in a published row, not enforcement, and the row should not make the two look like the same kind of fact.
• run_teardown verifies the child's start time before signalling. It is the only place this script signals anything, and pids on this box are cycling at ~1.5M in four days. The run traps are now installed before that read, so a signal arriving in the fork window cannot leak the lock and orphan the benchmark.
• --gate 0 is documented as unsatisfiable by construction — field 4 of /proc/loadavg counts the process doing the reading, so the floor is 1 on a perfectly idle box. Documented alongside it: the lock is the primary admission signal and the gate is secondary. A bounded good citizen on 4 of 32 cpus reads runnable 4–5 and trips --gate 3, while a single-threaded 100% hog reads ~1 and sails through. That revises guidance I gave the team earlier — still right against the loadavg EMA, wrong as an admission test.
• .hostlock-selftest* is gitignored; an aborted run left scratch files untracked.
• The test header no longer claims the suite "costs no measurable CPU". The R2 occupancy assertion deliberately spins 6 of 32 cpus for ~5 s, and someone running the suite next to a colleague's A/B on the strength of that sentence would inject exactly the contamination this tool exists to prevent.

Falsification

Tests 115 → 190. A conformance suite that cannot fail is worse than none, so every requirement is mutation-tested:

mutation	property	tests red
anchor defaults to $$ instead of $PPID	R1	17
runnable_now() returns a constant 1	R2	1
held_uid echoes the declared owner	R2	1
held_uid reports the reader's uid, not the holder's	R2	1
runnable_at_acquire re-read at print time	R2	2
STALE collapsed into EXPIRED	R3	8
acquire --wait expiry returns success	R4	3
grace narrowed back to zero-byte meta only	R5	3
UNPARSEABLE_GRACE 300 → 2	R5	2
reapable ignores a live anchor pid	R5	2
unverifiable takeover claims stale_pid	R5	1
status human branch keys off holder_alive	R5	2
status EXPIRED arm says the holder is gone	R5	2
live-anchor guard voids the holder's own ttl	R5	1
live-anchor guard drops the liveness test	R5	2
start-time comparison weakened to "the pid exists"	R5	2
zombie check [ "$state" != Z ] deleted	R6	2
state Z treated as dead unconditionally	R6	3
pkill -9 -f added to reap_if_dead	R7	2
timeout -s KILL smuggled into reap_if_dead	R7	1
run_teardown refuses every kill	R7	1
run_teardown check weakened to "the pid exists"	R7	1
run_teardown guard weakened with an `		`
fuser -k smuggled into reap_if_dead	R7	1
anti-theft guard accepts a zombie anchor (-d /proc)	R5/R6	1
lock_state drops the unverifiable-live arm	R3/R5	3
reaper calls a live unverifiable holder dead	R5	3
cmd_status STALE arm says FREE (nobody is here)	R3	3
pid_is_live treats every zombie as alive	R6	3
pid_is_live treats every state Z as dead	R6	3
/proc read restored to <file 2>/dev/null (stderr noise)	R2	2
cmd_run reads the start time before installing its traps	R7	1
(meta) one chk deleted from the suite	coverage	1

Baseline green at 208/208 in the same harness, immediately before and after every mutation. 33 mutations, all red.

Two mutations stay green, both recorded in the source rather than hidden, because a green mutation is otherwise indistinguishable from a missing test:

1. Dropping the -d /proc/<pid> check in holder_alive is semantically equivalent — proc_start_time already fails for a missing /proc entry. Not a coverage gap.
2. Flipping pid_is_live's unreadable-Threads: arm from "no evidence of death, keep the lock" toward theft. This one is a real gap and stays one. Producing a Z process with a readable stat and an unreadable status needs hidepid= or a lost microsecond race; the alternative is a HOSTLOCK_PROC hook in production, which would itself be a way to spoof liveness and steal a lock. What has changed since the last revision of this note is the argument for accepting it: it no longer rests on [ "" -le 1 ] exiting 2, which was true but incidental. The non-numeric case is now an explicit case arm that returns "alive" and says why, so the safe direction is written down rather than inherited from a test builtin's error behaviour — and, as a bonus, it no longer prints [: abc: integer expression expected on hostlock's stderr.

shellcheck clean on both files. No crate code; the suite is sleep-bound and holds no cores, apart from the one deliberate ~5 s six-thread burst that the R2 occupancy assertion needs.

Known gap, filed rather than smuggled in here: nothing runs this suite in CI — it is referenced by no workflow, and there is no shellcheck job. That is a separate concern from the conformance gap this PR closes, and it wants its own PR and its own argument about cost (the suite is ~3 minutes of mostly sleep).

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.29%. Comparing base (39b6b18) to head (16058dc).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1830      +/-   ##
==========================================
- Coverage   80.61%   80.29%   -0.33%     
==========================================
  Files         414      414              
  Lines      203362   203362              
  Branches   203362   203362              
==========================================
- Hits       163941   163287     -654     
- Misses      33869    34522     +653     
- Partials     5552     5553       +1     

Flag	Coverage Δ
cli-ort-linux	72.51% <ø> (ø)
cli-ort-windows	72.01% <ø> (ø)
mlas	85.10% <ø> (-0.14%)	⬇️
offline	80.42% <ø> (-0.34%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 12 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[justinchuby]

A question came up about whether run should keep ttl=0, prompted by a real incident: a genuinely-running aarch64 job that held ~1778% CPU for 5h40m after the work item motivating it had been abandoned — the owner moved HEAD off the branch at 03:28:16 and never looked back. The proposal was to pass an explicit finite --ttl on run for qemu jobs, on the reasoning that "a never-expiring lock held by a job whose owner has moved on is exactly the shape of what just happened".

I think run should keep ttl=0, and that a finite TTL there would make things worse rather than better. Laying out why, because the reasoning generalises past this script.

What ttl=0 on run actually guarantees

The rationale at :1015-1017 is exact:

run anchors to itself: that pid is exact and dies with the command on every exit path, so it needs no expiry.

Verified — cmd_run acquires, backgrounds the child, waits, and releases on the normal path (:891) and on INT/TERM/HUP (:883-885). So on the run path:

lock lifetime ≡ job lifetime. That is a true statement, and ttl=0 is what makes it true.

Why the incident does not argue against it

The incident was not a lock outliving its job. It was a job outliving its purpose while remaining perfectly alive. Had it been run under hostlock run, the lock would have been held for 5h40m and every second of that would have been accurate: the box really was occupied, by 18 cores of really-running work.

The lock was never the thing that was wrong. The job was.

Why a finite --ttl on run would be a regression

Set run --ttl 1800 against a 5-hour job and here is what you have built: at t=30min the lock becomes reapable while the job keeps burning 18 cores. The next acquirer takes it over — loudly, naming the holder (:394), which is good — and then starts measuring on a box that is 60% occupied by a process the lock no longer mentions.

That converts a true statement into a false one. The lock would report free while the host is anything but, which is a false-green of exactly the kind this repo has spent the day cataloguing. And it is the same observation made earlier in a different context — reclaiming the lock does not stop the load. A TTL bounds the claim; it cannot bound the work, and on the run path those are currently the same object, which is the property worth protecting.

--strict-reap (:64) blunts the edge by letting an acquirer refuse a host it had to reap, and --gate catches load with no lock behind it. But the default outcome of a finite run TTL is a lock that lies, and the default is what matters for a safety mechanism.

The mechanism that actually fits the failure

A job that outlives its purpose needs a bound on the job, not on the lock:

setsid timeout -k 30 5400 taskset -c 16-23 <cmd>

timeout kills the work; the run anchor then dies with it and the lock releases through the path it already has. Both statements stay true throughout, and the box is genuinely free at the moment the lock says so. That posture is already in use for qemu runs — taskset + --test-threads=2 + hard timeout + setsid + a pgrep -g reap check — and it is the correct fix for this failure mode. The lock never needed to change.

So: no TTL on run, and no per-call-site --ttl either. The instinct behind the ask was right — something must be bounded — but the thing to bound is the process, and it already is.

One thing genuinely worth having

acquire gets 3600 and run gets 0 (:1018-1023), matching the documented behaviour at :72. That asymmetry is correct and well-reasoned — acquire anchors to $PPID, a shell that can outlive the benchmark by days, so it does need a clock.

What is missing is any way to notice the case above from outside: a run lock held far longer than anything of its class should be. status prints the age (:648), so the information is present — but nothing draws attention to it. A soft advisory in status — held longer than N with ttl=0, printed as an observation and not a takeover — would have surfaced a 5h40m hold to anyone who looked, without giving anything the right to reap a live job. That keeps the honesty property intact and adds the only thing the incident actually lacked, which was visibility, not authority.