fix(OMN-20597): mirror knowledge-base-internal on the runner host and pin the fleet record to the refresh list

[jonahgabriel]

Summary

Ticket: OMN-20597 (child of OMN-16030).

knowledge-base-internal was the one high-volume self-hosted repo with no runner git mirror, so the job-started pre-seed logged [c2-mirror] no usable mirror ... leaving workspace cold and every job's actions/checkout fetched the whole 270 MB, 4,855-file tree from github.com.

• Window 2026-10-04T18:09Z to 2026-10-05T11:05Z, 643 self-hosted jobs: checkout minutes were knowledge-base-internal 2,048, omnibase_infra 19, omnimarket 8, omniclaude about 0.
• 551 knowledge-base-internal checkouts: p50 158 s, p90 442 s, max 1831 s. 13 of them ran 900 s or longer.
• Mirrored repos pre-seed in 2-7 s and check out in 0-4 s.

Change

• docker/runners/git-mirror-refresh.sh: add knowledge-base-internal to MIRROR_REPOS. The pre-seed in runner-job-started.sh keys off GITHUB_REPOSITORY, so no workflow changes.
• config/runner_fleet.yaml: add knowledge-base-internal to git_mirror.repos, and record omninode_infra. That repo has been in the script since OMN-19895 but was never added to the record.
• tests/unit/observability/runner_health/test_runner_fleet_config.py: a parity test now requires the yaml list to equal the script's MIRROR_REPOS, in order. A positive control proves the parser can see a difference. Before this change, origin/dev fails the parity check (script 10 entries, yaml 9).

Lab proof (omninode-pc, scratch dir /data/scratch/omn-20597, removed afterwards; no fleet mirror, daemon, runner or service touched)

The checkout's exact command was git -c protocol.version=2 fetch --no-tags --prune --no-recurse-submodules --depth=1 origin +<tip>:..., run against main tip a52dec5b at about 12:28Z:

case	seconds
cold, host	12.0
one-time git clone --mirror (138 MB)	13.6
seed a workspace from the local mirror, three commits stale	2.6
same depth-1 fetch from github.com after seeding	0.48
cold again, host	12.1
cold, inside a throwaway container from the live runner image (docker_default, --cpus 2, omnirunners.slice)	12.4

The quiet-hour cold fetch is 12 s right now. Jobs pay 110-160 s at p50, with stalls of 10-20 minutes, because the uplink is shared and contended. A seeded fetch moves a delta of a few KB, so jobs no longer depend on how busy the uplink is.

Delivery

This file reaches the host only through scripts/ci/check_runner_host_artifact_freshness.py --mode converge. That cron is not installed (ledger FRICTION 2026-09-28T13:15:06Z lane=runner-spread-83), and it is why the host copy still lists six repos. After merge, the lane runs --mode converge --only docker/runners/git-mirror-refresh.sh --ref origin/dev once. That replaces the file atomically and needs no container recreate and no service restart. The next 2-minute timer pass clones the five missing mirrors.

Test plan

• uv run pytest tests/unit/observability/runner_health/test_runner_fleet_config.py tests/scripts/test_git_mirror_refresh_maintenance_omn18802.py -q: 33 passed
• Before the change, the parity check fails on origin/dev
• bash -n docker/runners/git-mirror-refresh.sh, plus pre-commit on the changed files
• After merge: converge, then rerun one knowledge-base-internal job, comparing checkout step seconds before and after (OMN-20597 AC2/AC3)

Evidence-Ticket: OMN-20597
Evidence-Source: OCC#12923

[jonahgabriel]

Lab: head=f015bce4e397 host=h201 (omninode-pc) lane=ci-checkout-fix-9143 command=scratch probe in /data/scratch/omn-20597 (removed afterwards): exact actions/checkout depth-1 fetch of knowledge-base-internal main a52dec5b, cold vs seeded from a local --mirror clone that is 3 commits stale, plus the same cold fetch in a throwaway container from the live runner image (docker_default, --cpus 2, omnirunners.slice). observed=cold 12.0 s / 12.1 s on the host and 12.4 s in the container; mirror clone 13.6 s (138 MB); seed 2.6 s; seeded fetch 0.48 s. The same-hour before sample, a rerun at 12:32Z of 12 knowledge-base-internal main runs started together: checkout p50 119 s, max 138 s, 1,355 s in total, all logging the c2-mirror no-usable-mirror line. Neither the fleet mirror root nor the daemon nor any runner was touched.

[github-actions]

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: qwen3-review, gpt-oss-review
Models failed: none
New finding threads: 0
Deduped (already posted on this PR): 0
Below quorum (one model only, reported not threaded): 11
Nit-level findings suppressed: 1

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Below quorum: 11 finding(s) raised by one model only (OMN-18479)

These are reported and NOT dropped, but they get no thread and do not block: a single model's finding no other model reproduced is not evidence enough to stop a merge. Read them; act on them if they are right.

• [MAJOR] tests/unit/observability/runner_health/test_runner_fleet_config.py (qwen3-review) — Regex parser silently drops repos with inline comments | The helper _refresh_script_mirror_repos uses re.search with pattern r"^MIRROR_REPOS=(\n(.?)^)$" and re.M | re.S. The non-greedy .? combined
• [MAJOR] tests/unit/observability/runner_health/test_runner_fleet_config.py (qwen3-review) — Parity test compares ordered lists, but YAML and script may legitimately differ in order | The test test_runner_fleet_config_git_mirror_repos_match_refresh_script asserts list(config.git_mirror.repos)
• [MAJOR] config/runner_fleet.yaml: line 241, docker/runners/git-mirror-refresh.sh: line 52 (gpt-oss-review) — Parity test will always fail due to missing repo in script | The newly added test test_runner_fleet_config_git_mirror_repos_match_refresh_script asserts that the ordered list of repos in `config/run
• [MAJOR] config/runner_fleet.yaml and docker/runners/git-mirror-refresh.sh (gpt-oss-review) — Duplication of repository list across YAML and shell script | Maintaining the same list of mirrored repositories in both runner_fleet.yaml and git-mirror-refresh.sh creates a single point of incon
• [MINOR] tests/unit/observability/runner_health/test_runner_fleet_config.py (qwen3-review) — No test verifies that the YAML repos list has no duplicates | The parity test compares the YAML list to the script array. If both contain a duplicate entry, the test will pass but the mirror will fetc
• [MINOR] config/runner_fleet.yaml (qwen3-review) — New repo knowledge-base-internal is mirrored without visible access-control review | The diff adds knowledge-base-internal to the mirror fleet. The name suggests it contains internal or restricted con
• [MINOR] config/runner_fleet.yaml (qwen3-review) — Adding two repos to the serialized mirror extends the flock hold time | The YAML comment states that serialization is the entire point of the component: systemd Type=oneshot will not overlap runs, and
• [MINOR] tests/unit/observability/runner_health/test_runner_fleet_config.py (qwen3-review) — Positive-control test does not cover the case where MIRROR_REPOS is not found | The positive-control test test_refresh_script_mirror_repos_parser_sees_a_divergence only tests the happy path where the
• [MINOR] tests/unit/observability/runner_health/test_runner_fleet_config.py: lines 280‑285 (gpt-oss-review) — Regex used to parse MIRROR_REPOS is overly strict | The parser _refresh_script_mirror_repos uses the pattern ^MIRROR_REPOS=\(\n(.*?)^\)$ with multiline and dotall flags. This fails if there is any
• [MINOR] docker/runners/git-mirror-refresh.sh: line 53 (gpt-oss-review) — Mirroring knowledge-base-internal may increase network and storage load | Adding knowledge-base-internal to the mirror list will cause the mirror service to fetch a 270 MB repository each run, pot
• [MINOR] tests/unit/observability/runner_health/test_runner_fleet_config.py: lines 306‑311 (gpt-oss-review) — Parity test enforces order though order is semantically irrelevant | The test compares list(config.git_mirror.repos) to the parsed script list using equality, which fails if the two lists contain th

[github-actions]

✅ Hostile Reviewer — REVIEWED

Critical findings: 0
Major findings: 4
Total findings: 12
Models succeeded: qwen3-review,gpt-oss-review
Models unavailable: none

Action required: findings were posted as review threads. Address or reject each one, then resolve its thread — the Hostile Review Thread Gate fails while hostile-reviewer threads are unresolved (OMN-17492).

---

Semantics (OMN-17492 — the model finds, thread resolution gates)

Surface	Meaning	Blocks merge?
Review threads	Per-finding, posted by the reviewer	No (informational)
Hostile Review Thread Gate	Deterministic: unresolved hostile-reviewer threads exist	Fails until resolved (not yet a required context)
degraded verdict	Fewer than 2 models succeeded (infra)	Fails this job with a named reason

Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review: qwen3-review (Qwen3.8-27B), gpt-oss-review (gpt-oss-120b) (OMN-8468/OMN-8524/OMN-17492)