Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -170,8 +170,10 @@ jobs:
tests-herdr:
name: Behavior tests (Herdr)
runs-on: ubuntu-latest
# Real Herdr is slower than the portable suite; this is a hang tripwire,
# not the expected healthy end of the lane (estimate 15-40 min first cut).
# Healthy runs finish around 7 minutes. This job cap is a last-resort hang
# tripwire, not the expected end of the lane. The family-run step owns the
# tighter bound so a wedged suite fails fast with always() cleanup and
# timing artifacts still uploaded (docs/fm-test-portable-shards.md).
timeout-minutes: 75
steps:
- uses: actions/checkout@v6
Expand Down Expand Up @@ -252,6 +254,9 @@ jobs:
mkdir -p "$RUNNER_TEMP/fm-herdr"
bin/fm-herdr-ci-cleanup.sh snapshot "$RUNNER_TEMP/fm-herdr/sessions-before.json"
- name: Run real-Herdr family (serial, required)
# Comfortably above the ~7 min healthy wall and far below the 75 min
# job backstop. A hang must fail this step so cleanup still runs.
timeout-minutes: 20
run: |
set -eu
mkdir -p "$RUNNER_TEMP/fm-test"
Expand Down
11 changes: 6 additions & 5 deletions docs/fm-test-portable-shards.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,10 +105,11 @@ Portable shards, each portable serial shard, and the Herdr lane upload runner-ge

## Timeouts

| Job | timeout-minutes | Rationale |
|---|---:|---|
| portable parallel 1/2 | 10 | The measured shard sums are about three minutes and the timeout is a hang tripwire. |
| portable serial 1-4 | 15 | Each balanced shard is about five minutes, leaving roughly 3x hang-tripwire margin. |
| Herdr | 40 | The real-Herdr lane keeps its dedicated timeout. |
| Lane | Bound | Rationale |
|---|---|---|
| portable parallel 1/2 | job `timeout-minutes: 10` | The measured shard sums are about three minutes and the timeout is a hang tripwire. |
| portable serial 1-4 | job `timeout-minutes: 15` | Each balanced shard is about five minutes, leaving roughly 3x hang-tripwire margin. |
| Herdr | family-run step `timeout-minutes: 20`; job `timeout-minutes: 75` backstop | Healthy runs finish around 7 minutes, so the step bound is the hang tripwire (cleanup and timing artifacts still upload) while the job cap stays a last-resort backstop. |

Timeouts are hang tripwires rather than expected healthy durations.
`.github/workflows/ci.yml` owns the exact numbers.
35 changes: 35 additions & 0 deletions tests/fm-test-run.test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -627,6 +627,40 @@ SH
pass "jobs scheduler runs proven scripts; failure propagates; non-proven refused"
}

test_herdr_ci_family_run_has_a_step_timeout() {
# The required Herdr lane's hang tripwire is the family-run *step* bound, not
# the 75-minute job cap. Parse the workflow as YAML so nested `with.name`
# artifact keys cannot masquerade as the step contract.
command -v ruby >/dev/null 2>&1 \
|| fail "ruby is required to parse .github/workflows/ci.yml as YAML"
local json job_timeout step_timeout
json=$(ruby -ryaml -rjson -e '
doc = YAML.load_file(ARGV[0])
job = doc.fetch("jobs").fetch("tests-herdr")
step = job.fetch("steps").find { |s|
s.is_a?(Hash) && s["name"] == "Run real-Herdr family (serial, required)"
}
raise "missing family-run step" if step.nil?
raise "family-run step has no timeout-minutes" unless step.key?("timeout-minutes")
puts JSON.generate(
"job_timeout" => job.fetch("timeout-minutes"),
"step_timeout" => step.fetch("timeout-minutes")
)
' "$ROOT/.github/workflows/ci.yml") \
|| fail "could not parse tests-herdr timeouts from ci.yml"
job_timeout=$(python3 -c 'import json,sys; print(json.load(sys.stdin)["job_timeout"])' <<<"$json") \
|| fail "could not read job timeout from parsed workflow"
step_timeout=$(python3 -c 'import json,sys; print(json.load(sys.stdin)["step_timeout"])' <<<"$json") \
|| fail "could not read step timeout from parsed workflow"
[ "$job_timeout" = 75 ] \
|| fail "tests-herdr job backstop must stay 75 minutes, got $job_timeout"
[ "$step_timeout" = 20 ] \
|| fail "family-run step timeout must be 20 minutes, got $step_timeout"
[ "$step_timeout" -lt "$job_timeout" ] \
|| fail "family-run step timeout must be below the job backstop"
pass "Herdr CI family-run step times out at 20 min under a 75 min job backstop"
}

test_aggregate_json() {
local tmp a b
tmp=$(mktemp -d "${TMPDIR:-/tmp}/fm-test-run-aggjson.XXXXXX")
Expand Down Expand Up @@ -685,4 +719,5 @@ test_portable_serial_shards_partition_the_serial_lane
test_portable_serial_shard_lane_refusals
test_jobs_requires_proven_isolated
test_jobs_parallel_scheduler_and_failure_propagation
test_herdr_ci_family_run_has_a_step_timeout
test_aggregate_json
Loading