Repository navigation
fix(OMN-20598): bound a silent git HTTP transfer on the self-hosted runners so checkout retries instead of hanging - #4583
Conversation
…unners so checkout retries instead of hanging A depth-1 actions/checkout fetch hung 10-20 minutes per attempt before curl 56 on 2026-10-05 (job 111707578704: about 30 minutes in checkout). The job-started hook now adds http.lowSpeedLimit=1 / http.lowSpeedTime=60 to the job's GIT_CONFIG set through the shared accumulator it already flushes to GITHUB_ENV. Job-scoped, HTTP-only, env-overridable, fail-open on an invalid override. Picks up the staged work of lane rlane-ci-checkout-speed-9143-64f946.
|
Lab: head=d88ae41ad615 host=launching Mac (no lab host run) lane=ci-checkout-fix-9143 command=uv run pytest tests/scripts/test_runner_job_started_git_stall_bound.py tests/scripts/test_runner_job_started_mirror_rewrite.py tests/scripts/test_runner_job_started_hook_budget_omn18802.py -q observed=27 passed, 2 skipped (both behind the GNU realpath -m gate, so they run only on Linux). The silent-server fetch without the flushed settings did not finish inside 30 s; with them it aborts at the 3 s bound. A Linux run on h201 was abandoned after uv dependency resolution passed 10 minutes. The earlier lane rlane-ci-checkout-speed-9143-64f946 ran the pre-scrub version in the runner image on h101 and reported 13 passed. The change is not converged to any runner host. |
…12912) * evidence(OMN-20598): author OCC companion for OmniNode-ai/omnibase_infra#4583 OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head d88ae41ad6156ab55e9d13c3cdea8eb1a045a944. * evidence(OMN-20598): self-bind OCC#12912 + rebind contract_sha256 --------- Co-authored-by: omnimarket-bot <bot@omninode.ai>
There was a problem hiding this comment.
Hostile Reviewer — adversarial findings (OMN-17492)
Models succeeded: qwen3-review, gpt-oss-review
Models failed: none
New finding threads: 0
Deduped (already posted on this PR): 0
Below quorum (one model only, reported not threaded): 12
Nit-level findings suppressed: 1
The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.
Below quorum: 12 finding(s) raised by one model only (OMN-18479)
These are reported and NOT dropped, but they get no thread and do not block: a single model's finding no other model reproduced is not evidence enough to stop a merge. Read them; act on them if they are right.
- [MAJOR]
docker/runners/runner-job-started.sh:wire_git_transfer_stall_bound(qwen3-review) — Empty string override bypasses validation and applies default | The regex^0*[1-9][0-9]*$correctly rejects '0', but the testtest_invalid_override_fails_open_with_nothing_writtenexplicitly docum - [MAJOR]
docker/runners/runner-job-started.sh:wire_git_transfer_stall_bound(qwen3-review) — http.lowSpeedLimit=1 byte/s may abort legitimate slow transfers | The defaulthttp.lowSpeedLimitof 1 byte/s withhttp.lowSpeedTimeof 60s means any HTTP transfer that goes completely silent for 6 - [MAJOR]
tests/scripts/test_runner_job_started_git_stall_bound.py:test_flushed_bound_aborts_a_silent_fetch(qwen3-review) — Silent-server test may be flaky due to timing assumptions | The testtest_flushed_bound_aborts_a_silent_fetchuses a server that sends a partial response then goes silent, and asserts that the fetch - [MAJOR]
docker/runners/runner-job-started.sh: line ~380(gpt-oss-review) — Fail‑open on invalid low‑speed parameters disables protection | wire_git_transfer_stall_bound validates OMNI_GIT_LOW_SPEED_LIMIT and OMNI_GIT_LOW_SPEED_TIME but returns success (0) when validation fai - [MAJOR]
docker/runners/runner-job-started.sh: line ~380(gpt-oss-review) — Potential missing GIT_CONFIG_COUNT export after flush | c2_rewrite_flush writes the accumulated GIT_CONFIG_KEY/VALUE_ lines to GITHUB_ENV but does not emit a GIT_CONFIG_COUNT line. Git reads the cou - [MINOR]
tests/scripts/test_runner_job_started_git_stall_bound.py(qwen3-review) — No test for interaction with existing mirror rewrites | The testtest_full_script_success_path_flushes_stall_boundverifies that the stall bound is flushed, but does not test the interaction between - [MINOR]
docker/runners/runner-job-started.sh:wire_git_transfer_stall_bound(qwen3-review) — http.lowSpeedTime=60s may delay detection of true hangs | The defaulthttp.lowSpeedTimeof 60 seconds means that a truly hung transfer will not be detected until 60 seconds of silence have elapsed. - [MINOR]
docker/runners/runner-job-started.sh:wire_git_transfer_stall_bound(qwen3-review) — Stall bound applies to all git HTTP transfers, not just checkout | The comment states that the setting 'covers checkout, sibling clones and uv git fetches throughout the job.' This is a broad scope: a - [MINOR]
tests/scripts/test_runner_job_started_git_stall_bound.py(qwen3-review) — No test for GITHUB_ENV unwritable fail-open path | The_c2_rewrite_flushfunction has a fail-open path whenGITHUB_ENVis unwritable, but there is no test that verifies this behavior when the stal - [MINOR]
docker/runners/runner-job-started.sh: line ~350(gpt-oss-review) — Unvalidated key/value passed to shared accumulator | c2_env_add_config appends arbitrary strings to the GIT_CONFIG_KEY/VALUE_ arrays without sanitising them. If future callers supply untrusted input - [MINOR]
docker/runners/runner-job-started.sh: lines ~340‑360(gpt-oss-review) — Accumulator name no longer reflects its purpose | The array variable _C2_REWRITE_ENV_LINES is used to store both mirror‑rewrite settings and the new HTTP stall‑bound settings, yet its name suggests it - [MINOR]
docker/runners/runner-job-started.sh: line ~380(gpt-oss-review) — Accumulator not cleared after flush may cause duplicate entries | The flush function does not reset _C2_REWRITE_ENV_LINES or _C2_REWRITE_COUNT after writing to GITHUB_ENV. If _c2_rewrite_flush is invo
✅ Hostile Reviewer — REVIEWEDCritical findings: 0
Semantics (OMN-17492 — the model finds, thread resolution gates)
Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review: qwen3-review (Qwen3.8-27B), gpt-oss-review (gpt-oss-120b) (OMN-8468/OMN-8524/OMN-17492) |
Summary
Ticket: OMN-20598 (child of OMN-16030). This is the second-order fix next to OMN-20597 (omnibase_infra#4582, the knowledge-base-internal mirror).
On 2026-10-05 a depth-1
actions/checkoutfetch on the self-hosted fleet hung 10-20 minutes per attempt beforeRPC failed; curl 56 GnuTLS recv error (-54). Job 111707578704 (omninode-runner-15) spent about 30 minutes in checkout across three attempts, job 111694763319 (omnipc2-ci-runner-6) shows the same pattern, and 13 knowledge-base-internal checkouts ran 900 s or longer in 17 hours. Git sets no low-speed bound by default.Change
docker/runners/runner-job-started.sh:wire_git_transfer_stall_boundappendshttp.lowSpeedLimit=1andhttp.lowSpeedTime=60to the job'sGIT_CONFIG_*set through the shared accumulator. The C2b and C2c mirror rewrites already flush through that accumulator, so the job gets oneGIT_CONFIG_COUNTfor all of them. The two settings are job-scoped through GITHUB_ENV and affect onlyhttp(s), so git:// mirror traffic is untouched.OMNI_GIT_LOW_SPEED_LIMITandOMNI_GIT_LOW_SPEED_TIMEoverride them, and an invalid override writes neither setting and exits 0.Why 1 B/s over 60 s: upload-pack sends keepalives about every 5 s while the server generates a pack, so a slow server that is still working stays above the threshold. Only true silence trips the bound.
tests/scripts/test_runner_job_started_git_stall_bound.py(new) covers: the defaults are flushed, overrides are honoured, an invalid override fails open and writes nothing, the flushed settings abortls-remoteagainst a local server that goes silent mid-body within the bound, and the full script reaches the call on its success path (Linux only, GNUrealpath -m).tests/scripts/test_runner_job_started_mirror_rewrite.py: the full-script count assertion goes from 2 to 4, and its git subprocesses take a scrubbed env (OMN-18434 guard).uv run pytest tests/scripts/test_runner_job_started_git_stall_bound.py tests/scripts/test_runner_job_started_mirror_rewrite.py tests/scripts/test_runner_job_started_hook_budget_omn18802.py -qon macOS: 27 passed, 2 skipped (the GNU realpath gate). The original lane ran the same files on Linux in the runner image: 13 passed.RED control: the same silent-server fetch without the flushed settings did not finish inside the test's 30 s timeout. With the settings it aborts in about 3 s (
OMNI_GIT_LOW_SPEED_TIME=3).Pre-commit passes on the changed files.
Delivery
The hook is bind-mounted from
~/.omnibase/runners/docker/runners/on the runner hosts. After merge it reaches the hosts throughscripts/ci/check_runner_host_artifact_freshness.py --mode converge. No container recreate or service restart is needed. Not converged by this lane.Evidence-Ticket: OMN-20598
Evidence-Source: OCC#12912