Skip to content

fix(OMN-20598): bound a silent git HTTP transfer on the self-hosted runners so checkout retries instead of hanging - #4583

Merged
jonahgabriel merged 1 commit into
devfrom
jonah/omn-20598-git-http-stall-bound
Oct 5, 2026
Merged

jonahgabriel merged 1 commit into
devfrom
jonah/omn-20598-git-http-stall-bound

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Ticket: OMN-20598 (child of OMN-16030). This is the second-order fix next to OMN-20597 (omnibase_infra#4582, the knowledge-base-internal mirror).

On 2026-10-05 a depth-1 actions/checkout fetch on the self-hosted fleet hung 10-20 minutes per attempt before RPC failed; curl 56 GnuTLS recv error (-54). Job 111707578704 (omninode-runner-15) spent about 30 minutes in checkout across three attempts, job 111694763319 (omnipc2-ci-runner-6) shows the same pattern, and 13 knowledge-base-internal checkouts ran 900 s or longer in 17 hours. Git sets no low-speed bound by default.

Change

  • docker/runners/runner-job-started.sh: wire_git_transfer_stall_bound appends http.lowSpeedLimit=1 and http.lowSpeedTime=60 to the job's GIT_CONFIG_* set through the shared accumulator. The C2b and C2c mirror rewrites already flush through that accumulator, so the job gets one GIT_CONFIG_COUNT for all of them. The two settings are job-scoped through GITHUB_ENV and affect only http(s), so git:// mirror traffic is untouched. OMNI_GIT_LOW_SPEED_LIMIT and OMNI_GIT_LOW_SPEED_TIME override them, and an invalid override writes neither setting and exits 0.

  • Why 1 B/s over 60 s: upload-pack sends keepalives about every 5 s while the server generates a pack, so a slow server that is still working stays above the threshold. Only true silence trips the bound.

  • tests/scripts/test_runner_job_started_git_stall_bound.py (new) covers: the defaults are flushed, overrides are honoured, an invalid override fails open and writes nothing, the flushed settings abort ls-remote against a local server that goes silent mid-body within the bound, and the full script reaches the call on its success path (Linux only, GNU realpath -m).

  • tests/scripts/test_runner_job_started_mirror_rewrite.py: the full-script count assertion goes from 2 to 4, and its git subprocesses take a scrubbed env (OMN-18434 guard).

  • uv run pytest tests/scripts/test_runner_job_started_git_stall_bound.py tests/scripts/test_runner_job_started_mirror_rewrite.py tests/scripts/test_runner_job_started_hook_budget_omn18802.py -q on macOS: 27 passed, 2 skipped (the GNU realpath gate). The original lane ran the same files on Linux in the runner image: 13 passed.

  • RED control: the same silent-server fetch without the flushed settings did not finish inside the test's 30 s timeout. With the settings it aborts in about 3 s (OMNI_GIT_LOW_SPEED_TIME=3).

  • Pre-commit passes on the changed files.

Delivery

The hook is bind-mounted from ~/.omnibase/runners/docker/runners/ on the runner hosts. After merge it reaches the hosts through scripts/ci/check_runner_host_artifact_freshness.py --mode converge. No container recreate or service restart is needed. Not converged by this lane.

Evidence-Ticket: OMN-20598
Evidence-Source: OCC#12912

…unners so checkout retries instead of hanging

A depth-1 actions/checkout fetch hung 10-20 minutes per attempt before curl 56
on 2026-10-05 (job 111707578704: about 30 minutes in checkout). The job-started
hook now adds http.lowSpeedLimit=1 / http.lowSpeedTime=60 to the job's
GIT_CONFIG set through the shared accumulator it already flushes to GITHUB_ENV.
Job-scoped, HTTP-only, env-overridable, fail-open on an invalid override.
Picks up the staged work of lane rlane-ci-checkout-speed-9143-64f946.
@jonahgabriel

Copy link
Copy Markdown
Collaborator Author

Lab: head=d88ae41ad615 host=launching Mac (no lab host run) lane=ci-checkout-fix-9143 command=uv run pytest tests/scripts/test_runner_job_started_git_stall_bound.py tests/scripts/test_runner_job_started_mirror_rewrite.py tests/scripts/test_runner_job_started_hook_budget_omn18802.py -q observed=27 passed, 2 skipped (both behind the GNU realpath -m gate, so they run only on Linux). The silent-server fetch without the flushed settings did not finish inside 30 s; with them it aborts at the 3 s bound. A Linux run on h201 was abandoned after uv dependency resolution passed 10 minutes. The earlier lane rlane-ci-checkout-speed-9143-64f946 ran the pre-scrub version in the runner image on h101 and reported 13 passed. The change is not converged to any runner host.

jonahgabriel pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Oct 5, 2026
…12912)

* evidence(OMN-20598): author OCC companion for OmniNode-ai/omnibase_infra#4583

OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head d88ae41ad6156ab55e9d13c3cdea8eb1a045a944.

* evidence(OMN-20598): self-bind OCC#12912 + rebind contract_sha256

---------

Co-authored-by: omnimarket-bot <bot@omninode.ai>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: qwen3-review, gpt-oss-review
Models failed: none
New finding threads: 0
Deduped (already posted on this PR): 0
Below quorum (one model only, reported not threaded): 12
Nit-level findings suppressed: 1

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Below quorum: 12 finding(s) raised by one model only (OMN-18479)

These are reported and NOT dropped, but they get no thread and do not block: a single model's finding no other model reproduced is not evidence enough to stop a merge. Read them; act on them if they are right.

  • [MAJOR] docker/runners/runner-job-started.sh:wire_git_transfer_stall_bound (qwen3-review) — Empty string override bypasses validation and applies default | The regex ^0*[1-9][0-9]*$ correctly rejects '0', but the test test_invalid_override_fails_open_with_nothing_written explicitly docum
  • [MAJOR] docker/runners/runner-job-started.sh:wire_git_transfer_stall_bound (qwen3-review) — http.lowSpeedLimit=1 byte/s may abort legitimate slow transfers | The default http.lowSpeedLimit of 1 byte/s with http.lowSpeedTime of 60s means any HTTP transfer that goes completely silent for 6
  • [MAJOR] tests/scripts/test_runner_job_started_git_stall_bound.py:test_flushed_bound_aborts_a_silent_fetch (qwen3-review) — Silent-server test may be flaky due to timing assumptions | The test test_flushed_bound_aborts_a_silent_fetch uses a server that sends a partial response then goes silent, and asserts that the fetch
  • [MAJOR] docker/runners/runner-job-started.sh: line ~380 (gpt-oss-review) — Fail‑open on invalid low‑speed parameters disables protection | wire_git_transfer_stall_bound validates OMNI_GIT_LOW_SPEED_LIMIT and OMNI_GIT_LOW_SPEED_TIME but returns success (0) when validation fai
  • [MAJOR] docker/runners/runner-job-started.sh: line ~380 (gpt-oss-review) — Potential missing GIT_CONFIG_COUNT export after flush | c2_rewrite_flush writes the accumulated GIT_CONFIG_KEY/VALUE_ lines to GITHUB_ENV but does not emit a GIT_CONFIG_COUNT line. Git reads the cou
  • [MINOR] tests/scripts/test_runner_job_started_git_stall_bound.py (qwen3-review) — No test for interaction with existing mirror rewrites | The test test_full_script_success_path_flushes_stall_bound verifies that the stall bound is flushed, but does not test the interaction between
  • [MINOR] docker/runners/runner-job-started.sh:wire_git_transfer_stall_bound (qwen3-review) — http.lowSpeedTime=60s may delay detection of true hangs | The default http.lowSpeedTime of 60 seconds means that a truly hung transfer will not be detected until 60 seconds of silence have elapsed.
  • [MINOR] docker/runners/runner-job-started.sh:wire_git_transfer_stall_bound (qwen3-review) — Stall bound applies to all git HTTP transfers, not just checkout | The comment states that the setting 'covers checkout, sibling clones and uv git fetches throughout the job.' This is a broad scope: a
  • [MINOR] tests/scripts/test_runner_job_started_git_stall_bound.py (qwen3-review) — No test for GITHUB_ENV unwritable fail-open path | The _c2_rewrite_flush function has a fail-open path when GITHUB_ENV is unwritable, but there is no test that verifies this behavior when the stal
  • [MINOR] docker/runners/runner-job-started.sh: line ~350 (gpt-oss-review) — Unvalidated key/value passed to shared accumulator | c2_env_add_config appends arbitrary strings to the GIT_CONFIG_KEY/VALUE_ arrays without sanitising them. If future callers supply untrusted input
  • [MINOR] docker/runners/runner-job-started.sh: lines ~340‑360 (gpt-oss-review) — Accumulator name no longer reflects its purpose | The array variable _C2_REWRITE_ENV_LINES is used to store both mirror‑rewrite settings and the new HTTP stall‑bound settings, yet its name suggests it
  • [MINOR] docker/runners/runner-job-started.sh: line ~380 (gpt-oss-review) — Accumulator not cleared after flush may cause duplicate entries | The flush function does not reset _C2_REWRITE_ENV_LINES or _C2_REWRITE_COUNT after writing to GITHUB_ENV. If _c2_rewrite_flush is invo

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

✅ Hostile Reviewer — REVIEWED

Critical findings: 0
Major findings: 5
Total findings: 13
Models succeeded: qwen3-review,gpt-oss-review
Models unavailable: none

Action required: findings were posted as review threads. Address or reject each one, then resolve its thread — the Hostile Review Thread Gate fails while hostile-reviewer threads are unresolved (OMN-17492).


Semantics (OMN-17492 — the model finds, thread resolution gates)

Surface Meaning Blocks merge?
Review threads Per-finding, posted by the reviewer No (informational)
Hostile Review Thread Gate Deterministic: unresolved hostile-reviewer threads exist Fails until resolved (not yet a required context)
degraded verdict Fewer than 2 models succeeded (infra) Fails this job with a named reason

Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review: qwen3-review (Qwen3.8-27B), gpt-oss-review (gpt-oss-120b) (OMN-8468/OMN-8524/OMN-17492)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant