Repository navigation
fix(bin): keep process identity stable across host clock steps - #45
Merged
Merged
Conversation
…ross clock steps The pending-reply recovery sender check and the Herdr lab viewer ownership check identified processes by ps lstart text, which on Linux is the wall-clock-derived boot time plus start ticks. A host clock step (WSL2 steps about every 30 seconds) re-renders it, so a live recovery sender read as dead and a running lab viewer pair read as not owned. Both now use /proc/<pid>/stat start ticks where readable, like fm_pid_identity and task_process_identity, and keep the ps form elsewhere. Records already written in the legacy lstart form are still compared through ps, so an upgrade does not strand an in-flight recovery or a running viewer.
This was referenced Sep 29, 2026
knowttl
added a commit
that referenced
this pull request
Sep 30, 2026
* fix(bin): keep process identity stable across host clock steps (#45) * fix(bin): keep pending-reply sender and lab viewer identity stable across clock steps The pending-reply recovery sender check and the Herdr lab viewer ownership check identified processes by ps lstart text, which on Linux is the wall-clock-derived boot time plus start ticks. A host clock step (WSL2 steps about every 30 seconds) re-renders it, so a live recovery sender read as dead and a running lab viewer pair read as not owned. Both now use /proc/<pid>/stat start ticks where readable, like fm_pid_identity and task_process_identity, and keep the ps form elsewhere. Records already written in the legacy lstart form are still compared through ps, so an upgrade does not strand an in-flight recovery or a running viewer. * no-mistakes(document): Document clock-stable process identity and legacy records * fix(bin): prevent Linux remote job worker pileups after clock changes (#46) * fix(bin): keep Linux remote job worker identity stable across clock steps The remote job worker identified its own processes (lock owner, staging owner, job claims, lanes, command groups) by `ps -o lstart=` text. On Linux, procps renders lstart from the current boot time, which moves whenever the wall clock is stepped (NTP, VM or WSL2 time sync, resume). After a step a healthy worker no longer matched its own lock record, so every remote call started another detached supervisor beside it, the losers restarted for minutes, the serving loop blocked on live lanes it thought had exited, a competing worker reclaimed the live lock, and running jobs were published as "remote job worker stopped before this job completed". - Record Linux process identity as starttime=<stat field 22>, which no clock step moves; Darwin keeps ps lstart, unchanged. - Keep records written by earlier workers comparable: an lstart record is compared as lstart, and a Linux lock owner still recorded as lstart is identified by pid and exact command, so an update replaces it in place and drains supervisors already piled beside it instead of stranding it. - A serving worker that has lost its ownership lock now stops its own active execution and exits on a stop signal instead of re-arming, and never writes quarantine into a lock it does not own. * no-mistakes(document): Document Linux remote worker identity and shutdown behavior
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
can you explicitly verify that it is an actual bug against the source code then implement a fix so we can also create a pull request for it in upstream?
also ensure that before you create a pull request on the upstream repo please also ensure we properly follow the upstream repo contribution rules if there are any and the standards it has defined?
also ensure that the fixes generalize across the firstmate repo.
two pull requests
Context the ask refers to: a scout confirmed that firstmate's Linux remote-job worker identified processes by the text of
ps -o lstart=, which on Linux is the kernel's wall-clock-derived boot time plus start ticks, so every host clock step (WSL2 steps about every 30 seconds) changes every running process's recorded start and a live process stops matching its recorded identity.The fix for the remote-job worker itself is the first of the two pull requests and is a separate task.
To make the fix generalize, the scout swept
bin/for the same flaw and found two more vulnerable sites (code-read verdicts, not reproduced):bin/fm-pending-reply-lib.sh(fm_pending_reply_pid_identity/fm_pending_reply_sender_alive, around lines 979-993), where after a clock step a live recovery sender reads as dead; and the Herdr lab viewer/launcher ownership check inbin/fm-herdr-lab.sh(around lines 202-231) andbin/fm-herdr-lab-viewer.py(around line 83), where after a step the pair reads as not owned.Sites already safe for reference:
fm_pid_identityinbin/fm-wake-lib.sh(the July watcher fix, upstream kunchenguid#752),task_process_identityinbin/fm-teardown.sh, andpidIdentityinbin/fm-extension.mjs, which all use/proc/<pid>/statstart ticks.This task is the second pull request: fix those two remaining sites the same way, with compatibility for records already written in the old
lstartform.The scout report's section 9 D recommends exactly this: switch those two sites to
/proc/<pid>/statstart-tick identity where readable, keeppselsewhere, keep recognizing legacylstartrecords, and pin each with a regression test that changes boot time but not start ticks.What Changed
/procstart ticks for recovery sender identity while preserving command identity and apsfallback.lstartrecords.Risk Assessment
✅ Low: The change is limited to the two reported identity checks, preserves comparison of legacy records, and adds behavioral regressions for clock steps.
Testing
Both focused test scripts passed, covering simulated clock steps, PID reuse, and legacy records. A real named Herdr lab viewer attached with start-tick identities, stopped, and was torn down. An initial lab attempt hit the helper’s duplicate-tripwire guard because provision already performs prepare; the fresh provision-only run passed. No real host clock was changed.
Evidence: Real Herdr lab viewer lifecycle
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
tests/fm-pending-reply.test.shtests/fm-herdr-lab.test.shbin/fm-herdr-lab.sh provision <named-lab>bin/fm-herdr-lab.sh viewer start <named-lab>bin/fm-herdr-lab.sh viewer stop <named-lab>bin/fm-herdr-lab.sh teardown <named-lab>✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.