ci: bound Herdr behavior shard with a 20-minute step timeout

[kunchenguid]

Intent

Bound firstmate's Herdr behavior CI shard with a step-level timeout so a hung suite fails fast with evidence instead of occupying a runner until the 75-minute job cap.

From the 2026-08-14 fleet CI audit (captain-approved to dispatch): firstmate's Behavior tests (Herdr) CI job relies on a 75-minute JOB cap (added in firstmate#2191) as its only stop - but healthy runs finish in ~7 min, so 75 min is a HANG TRIPWIRE, not a target: a wedged shard holds a runner for the full 75 min before cancel (e.g. run 31524483885, ~92-min wall).

Fix: add a STEP-LEVEL timeout INSIDE the Herdr behavior shard/script so a hung shard FAILS FAST with evidence instead of occupying a runner to the 75-min cap. Choose a bound comfortably above the ~7-min healthy time but far below 75 min. Keep the job cap as a backstop. Do NOT touch the 479 action_required runs - those are fork-PR approvals, not flakes, and out of scope.

This edits firstmate's shared CI / Herdr material. Ship no-mistakes +yolo; merge on green. Do not auto-fix unrelated flaky tests such as tests/fm-procevent.test.sh.

What Changed

• Add a 20-minute timeout-minutes on the required Herdr family-run CI step so a hung suite fails that step (cleanup and timing artifacts still upload) instead of occupying a runner until the 75-minute job cap.
• Keep the 75-minute job timeout as a last-resort backstop and document the two-level Herdr bound in the portable-shards timeout table.
• Add a YAML-parsed regression test that asserts the family-run step timeout is 20 minutes and the job backstop stays 75.

Risk Assessment

✅ Low: The Herdr family-run step is now bounded at 20 minutes under the kept 75-minute job backstop, which is the requested fail-fast contract and leaves always() cleanup and artifact upload intact.

Testing

Parsed the real tests-herdr workflow as YAML, reproduced the missing step timeout on the base commit, confirmed the official contract test passes after the fix, and simulated a wedged family-run so a hang now fails at 20 minutes with cleanup and timing artifacts still uploaded while the 75-minute job cap stays the backstop.

Evidence: Reviewer-facing hang-tripwire comparison (base vs target)

Source: Reviewer-facing hang-tripwire comparison (base vs target)

# Herdr behavior shard: step-level hang tripwire

End-user of this change: a CI operator watching a wedged `Behavior tests (Herdr)` job.

## What the runner will do

Parsed `.github/workflows/ci.yml` as YAML (Psych / the same document GitHub Actions consumes). Not a text grep.

| | Base `6789876` | Target `221aebd` |
|---|---|---|
| Job `tests-herdr` `timeout-minutes` | 75 | 75 (kept as backstop) |
| Family-run step `timeout-minutes` | **missing** | **20** |
| Cleanup `if:` | `always()` | `always()` |
| Upload timing/diagnostics `if:` | `always()` | `always()` |

GitHub Actions semantics (docs.github.com workflow syntax):

- **Step** `timeout-minutes`: "The maximum number of minutes to run the step before killing the process."
- **Job** `timeout-minutes`: "The maximum number of minutes to let a job run before GitHub automatically cancels it."

A step kill leaves the job running, so the two `if: always()` steps after the family-run still execute. A job-cap cancel is the whole job stopping — that is the 75-minute hang the audit called out (run 31524483885).

20 minutes is above the ~7-minute healthy wall and far below the 75-minute backstop.

## Contract test (official suite function)

`tests/fm-test-run.test.sh::test_herdr_ci_family_run_has_a_step_timeout`

- Against **base** workflow: `not ok - could not parse tests-herdr timeouts from ci.yml` (`family-run step has no timeout-minutes`). Exit 1.
- Against **target** workflow: `ok - Herdr CI family-run step times out at 20 min under a 75 min job backstop`. Exit 0.

The reported failure exists before the fix and is gone after it.

## Hung-suite experience (scaled clock)

1 CI minute = 80 ms wall so a 75-minute occupation is observable without burning a runner.

`` `
BASE (job cap only)
  t=    0 min  family-run starts and wedges
  t=   75 min  job timeout-minutes=75 cancels the whole job (only stop)
  t=   75 min  cleanup/upload not a reliable evidence path
  => runner occupied 75 CI minutes; evidence_uploaded=false

TARGET (this change)
  t=    0 min  family-run starts and wedges
  t=   20 min  step timeout-minutes=20 fires; hung family-run is killed
  t= 20.5 min  Cleanup job-owned Herdr lab sessions ran (if: always())
  t= 21.0 min  Upload Herdr timing and diagnostics ran (if: always())
  => runner occupied 20 CI minutes; evidence_uploaded=true
`` `

A hang now fails ~55 runner-minutes sooner, with cleanup and the timing/diagnostics artifact still uploaded. The 75-minute job cap remains the last-resort backstop.

## Scope (forbidden work)

Changed files vs base: `.github/workflows/ci.yml`, `docs/fm-test-portable-shards.md`, `tests/fm-test-run.test.sh`.

- `tests/fm-procevent.test.sh` not touched.
- Diff does not mention `action_required` (the 479 fork-PR approvals stay out of scope).

Evidence: Scaled hung-suite timeline: 75-minute job cancel vs 20-minute step fail-fast with evidence

Source: Scaled hung-suite timeline: 75-minute job cancel vs 20-minute step fail-fast with evidence

BASE (job cap only) t= 0 min family-run starts and wedges t= 75 min job timeout-minutes=75 cancels the whole job (only stop) => runner occupied 75 CI minutes; evidence_uploaded=False TARGET (this change) t= 0 min family-run starts and wedges t= 20 min step timeout-minutes=20 fires; hung family-run is killed t= 20.5 min Cleanup job-owned Herdr lab sessions ran (if: always()) t= 21.0 min Upload Herdr timing and diagnostics ran (if: always()) => runner occupied 20 CI minutes; evidence_uploaded=True

Herdr hang tripwire — compressed end-user simulation
Time scale: 1 CI minute = 0.08s wall (so a 75-min occupation is observable)

BASE (before the change): tests-herdr has only timeout-minutes: 75 on the job.
A wedged family-run holds the runner until that job cap. Cleanup/upload are
not a reliable evidence path once the job itself is cancelled.

  t=    0 min  BASE (job cap only): family-run starts and wedges (suite never returns)
  t=   75 min  job timeout-minutes=75 cancels the whole job (only stop)
  t=   75 min  job cancelled at cap; cleanup/upload not a reliable evidence path
  => runner occupied 75 CI minutes; evidence_uploaded=False

TARGET (this change): family-run step timeout-minutes: 20; job stays 75.
A wedged family-run is killed at 20 minutes. The job is still running, so
if: always() cleanup and artifact upload still execute.

  t=    0 min  TARGET (step tripwire + job backstop): family-run starts and wedges (suite never returns)
  t=   20 min  step timeout-minutes=20 fires; hung family-run is killed
  t= 20.5 min  Cleanup job-owned Herdr lab sessions ran (if: always())
  t= 21.0 min  Upload Herdr timing and diagnostics ran (if: always())
  => runner occupied 20 CI minutes; evidence_uploaded=True

Saved ~55 runner-minutes on a hang. Job cap remains the last-resort backstop.

Evidence: Official contract test on the target workflow

Source: Official contract test on the target workflow

ok - Herdr CI family-run step times out at 20 min under a 75 min job backstop exit=0

repo=/Users/kunchen/.no-mistakes/worktrees/016d88035d58/01M01T1AM9F508XFJAHTK1GN6S
2026-08-15T04:28:24Z
ok - Herdr CI family-run step times out at 20 min under a 75 min job backstop
exit=0

Evidence: Official contract test on the base workflow (must fail)

Source: Official contract test on the base workflow (must fail)

family-run step has no timeout-minutes (RuntimeError) not ok - could not parse tests-herdr timeouts from ci.yml exit=1

repo=/var/folders/0k/bf8mwt2n5qddzk24r20gfk0c0000gn/T/no-mistakes-evidence/01M01T1AM9F508XFJAHTK1GN6S/base-tree
2026-08-15T04:28:24Z
-e:8:in `<main>': family-run step has no timeout-minutes (RuntimeError)
not ok - could not parse tests-herdr timeouts from ci.yml
exit=1

Evidence: Parsed tests-herdr timeout model (target)

Source: Parsed tests-herdr timeout model (target)

{
  "job": "tests-herdr",
  "job_timeout_minutes": 75,
  "family_run_step": {
    "name": "Run real-Herdr family (serial, required)",
    "timeout_minutes": 20,
    "has_timeout": true
  },
  "cleanup_step": {
    "name": "Cleanup job-owned Herdr lab sessions",
    "if": "always()",
    "runs_after_step_failure_or_cancel": true
  },
  "upload_step": {
    "name": "Upload Herdr timing and diagnostics",
    "if": "always()",
    "runs_after_step_failure_or_cancel": true
  },
  "contract": {
    "job_backstop_minutes": 75,
    "step_tripwire_minutes": 20,
    "step_is_tighter_than_job": true,
    "healthy_runs_about_minutes": 7,
    "step_above_healthy_runtime": true,
    "step_far_below_job_cap": true,
    "cleanup_and_upload_survive_step_timeout": true
  }
}

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

• ruby parse-herdr-timeouts.rb semantic YAML model of .github/workflows/ci.yml jobs.tests-herdr (target 221aebd vs base 6789876)
• bash test_herdr_ci_family_run_has_a_step_timeout.sh against the target workflow — ok - Herdr CI family-run step times out at 20 min under a 75 min job backstop
• bash test_herdr_ci_family_run_has_a_step_timeout.sh against the base workflow — not ok / family-run step has no timeout-minutes (regression exists before the fix)
• python3 simulate-herdr-hang.py scaled hung-suite timeline: base holds the runner 75 CI minutes with no reliable evidence path; target kills the step at 20 minutes and still runs if: always() cleanup + artifact upload
• scope check that the diff does not touch tests/fm-procevent.test.sh or action_required

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.