Skip to content

Sweep runner: wait no longer reports a just-finished job as not running (flaky required check) - #576

Merged
seathatflowsinourveins merged 1 commit into
mainfrom
claude/sweep-runner-wait-flake-20261001
Oct 1, 2026
Merged

seathatflowsinourveins merged 1 commit into
mainfrom
claude/sweep-runner-wait-flake-20261001

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Owner

Scope

  • What this PR changes, in one or two sentences: fixes a check-then-act race in the landscape sweep runner's wait (a job that finished between its done check and its lock check was reported as "not running"), adds a deterministic test and failure diagnostics, and records two proven mistakes in the anti-pattern log.
  • Base commit: b72c588f
  • Lane: lane:foundation
  • Owned paths touched: tools/sota-convergence/landscape-sweep/codex_job.py, tests/test_landscape_sweep_harness.py, docs/harness-defaults.md, manifests/evidence.json (hot-file commit, last).

SOTA sources

  • flock(2), Linux man-pages (https://man7.org/linux/man-pages/man2/flock.2.html): a lock is released when its last descriptor is closed, so a runner that writes its completion marker before it closes the lock lets a reader decide from the marker once the lock is free.
  • This repository's runner: tools/sota-convergence/landscape-sweep/codex_job.py (finish writes done before run closes the job lock; running probes the lock). No upstream tool is replaced; the change is three lines in our own wrapper around codex exec.
  • The anti-pattern log's contract: docs/harness-defaults.md, "Upstream verification and compounding learning" (AGENTS.md: record each proven mistake).

Evidence-class table

Claim Evidence class Command / receipt
A job that finishes during the lock check is reported as done synthetic (fixture: the lock check itself completes the job) python3 -m unittest tests.test_landscape_sweep_harness.WaitFinishRaceTests; failing first on the unfixed runner: 'done exit=none (the job is not running)' != 'done exit=0'
A job that never ran is still reported as not running synthetic the same class, second test
The hosted failure on PR #360 (run 36836458360) was this race inference: the run kept no output of wait; a single-core stress loop did not reproduce it before the fix the test now prints wait's output and the runner's files on failure
The harness's other tests still pass local_integration python3 -m unittest tests.test_landscape_sweep_harness tests.test_adoption_docs_consistency

Local commands run

$ python3 -m unittest tests.test_landscape_sweep_harness.WaitFinishRaceTests     (test added, runner unfixed)
FAILED (failures=1)   AssertionError: 'done exit=none (the job is not running)' != 'done exit=0'
$ python3 -m unittest tests.test_landscape_sweep_harness tests.test_adoption_docs_consistency
Ran 209 tests in 48.440s   OK (skipped=4)   (exit 0)
$ python3 scripts/validate.py
{"components": 69, "hashed_files": 8720, "profiles": 4, "receipts": 180, "status": "passed"}   (exit 0)
$ git diff --check origin/main HEAD
(no output, exit 0)

Decision record

None: a defect repair. The two anti-pattern rows in docs/harness-defaults.md record what happened and the rule that prevents it (this race; and the evidence-manifest rows that a hot-file re-run of PR #548 dropped twice on 2026-10-01).

Host evidence

Not applicable.

Checklist

  • New/changed GitHub Actions are pinned to a full commit SHA with a version comment (none changed).
  • New/changed workflows declare top-level permissions: contents: read (none changed).
  • No secrets are printed, logged or committed; no new required secret was added without a documented owner.
  • No new paid hosting, subscription or billing surface was introduced.
  • Peer-owned untracked files and worktrees were preserved (not deleted, moved or overwritten).

🤖 Generated with Claude Code

@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Oct 1, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-01T09:19:19.893251Z 2414ee8 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

…as done, not as "not running"

`codex_job.py wait` read `done`, then the job lock. A job that finished between the two reads was reported as
"done exit=none (the job is not running)". The runner writes `done` before it releases the lock (flock(2): the lock
goes when its last descriptor closes), so wait now reads `done` again once it sees the lock free.

- tools/sota-convergence/landscape-sweep/codex_job.py: the re-read in `wait`.
- tests/test_landscape_sweep_harness.py: `WaitFinishRaceTests` (synthetic fixture: the lock check itself completes
  the job; fails before the fix with 'done exit=none (the job is not running)' != 'done exit=0'; a job that never
  ran is still reported as not running); `RunnerCase.diagnose`, used by the web-search-modes test so a failed start
  or wait prints the command's output and the runner's files.
- docs/harness-defaults.md: two anti-pattern rows (this race; the evidence-manifest rows a hot-file re-run dropped).

Observed once: hosted run 36836458360 on PR #360 failed test_web_search_modes_are_staged_bound_and_recorded at its
wait step, with no output retained. The fixture shows the mechanism; a single-core stress loop did not reproduce
the hosted failure before the fix, so the match with that run is an inference.

Tests: python3 -m unittest tests.test_landscape_sweep_harness tests.test_adoption_docs_consistency -> 209 tests OK
(4 skipped), exit 0.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@seathatflowsinourveins
seathatflowsinourveins force-pushed the claude/sweep-runner-wait-flake-20261001 branch from 2414ee8 to 330be78 Compare October 1, 2026 09:36
@seathatflowsinourveins
seathatflowsinourveins merged commit 2fd9b31 into main Oct 1, 2026
25 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/sweep-runner-wait-flake-20261001 branch October 1, 2026 10:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant