Skip to content

CI policy: 30-minute job timeout (workflow-layer policy over the modeled GitHub default) - #6154

Merged
briansrls merged 46 commits into
mainfrom
session/sunny-newt-884
Jul 2, 2026
Merged

briansrls merged 46 commits into
mainfrom
session/sunny-newt-884

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Operator directive (2026-07-02): CI jobs now time out at 30 minutes instead of inheriting GitHub's 360-minute platform default.

Why: a healthy floor is ~9 min solo. Observed today: floor jobs grinding 6+ hours single-threaded (one claim_executor at 8h57m wall / 8h07m CPU on one core), each squatting a runner slot until the 6h platform ceiling — a self-sustaining queue deadlock across all 25 fleet runners while the boxes sat ~87% idle. A >30-min job is pathological, not legitimately slow; it must fail fast and release its slot, surfacing the pathology instead of hiding it.

Layering (§3): extdeps/github/actions.dag's default_job_timeout_minutes = 360 is deliberately unchanged — it faithfully models GitHub's real platform default. The new gunbc_ci_job_timeout_policy_minutes = 30 is the workflow-layer policy on top, with a Terminal disposition carrying the rationale.

Receipts: ci.yml regenerated via generated_artifact_gate main_wet (diff is exactly 360→30 on the three jobs); run_generated_artifact_drift_gate_body PASS wet.

Companion work this wave: single-threaded resolver pathology (the actual reason jobs exceed 9 min) is being dispatched separately.

🤖 Generated with Claude Code

briansrls and others added 30 commits July 1, 2026 20:31
- §2 posture: required CI back on the fleet (#6111 revert; Ubicloud detour
  bought no wall-clock — serial compile-clean wall dominates); milestones
  updated; fleet-return group becomes fleet-hardening (the return happened
  ungated, the gates are now owed).
- New: 2-compile-clean-serial (the ~47min width-immune dominant cost +
  its three levers), 2-runner-shape (operator 1:3 core:GiB + swap slot
  contract), cap-deconflation corrected (width=9 was live by construction;
  the deferred item was only the concept rename), CD-transport reframed
  post-revert, G4 gains the dup-run + lingering in_progress evidence.
- §3 audit: 3-audit-artifact-freshness (operator freshness-gate ask; the
  .gitignore landed-drift incident as the RED receipt, closes via the
  merge-admission freshness block) and 3-audit-gate-disagreement
  (compile-clean green while discovery red on the same tree — the
  extdeps.shell collision; one tree, one verdict).
- §1 5-defork gains the live evidence line (the shell fork kept main red).

Witnesses re-targeted in lockstep; verified by execution: main_wet regen,
drift gate, roadmap_authority_witnesses, doc-graph orphans+dangling all PASS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d MVP

Applies the operator's three directives in one lockstep batch:

1. Roadmap-review blockers:
   - Ubicloud tone: detour was the diagnostic + emergency valve, not a
     failure; stays the break-glass/burst option.
   - CD transport reframed as target-host proof (LocalShell valid only
     when runner identity proves host=srv1), not Ubicloud-specific.
   - v1-collapse dependency edges now REAL: roadmap_spawner readiness is
     all-parents (node_dep_done folds every edge, not the first), with a
     RED/GREEN multi-parent witness; 5-collapse-v1 gains edges from
     dissolve-patches, test-migration, seed-honesty, defork.
   - workload-class admission model + cap/admission de-conflation (three
     facts, one knob today) + sound host admission (Guaranteed vs Burst
     with Σ-accounting receipts) + live host read seam as first-class
     nodes; G1 stale "dormant on Ubicloud" conditional fixed (cursor
     review finding on #6110).
   - Floor group renamed: throughput — reduce work, then schedule it.
   - Shelf: privacy/isolation model + structural correctness walls.

2. Fabric dispatch design (operator-signed): core design rule prose
   (ProviderOffer adapter / RunShape→Allocation→Receipt / strict default /
   oversubscription earned by receipts), StrictLease Policy 0, CI shape
   labels (runs-on as a projection; runner availability is backpressure),
   ProviderOffer rows + dormant-Ubicloud design-break probe, BurstLease
   Policy 1; TERMINAL reworded to "required CI runs on the compute
   fabric"; converge re-land gains the narrow srv2 first-acceptance case;
   ordering prose ①–⑩.

3. Stateless frontend MVP on fabric: first non-CI product-shaped
   consumer (SiteArtifact/WebServiceShape/DomainRoute/ServiceAllocation,
   manual DNS, digest-proving receipt, milestones A/B), service
   allocation receipt under control plane, fabric-hostable rule prose,
   shelf HTML line scoped.

Verified by execution: main_wet regen, drift gate, roadmap_authority
witnesses (new fabric pins + updated interleaving neighbor), spawner
witnesses (incl. new all-parents RED control), doc-graph orphans +
dangling links — all PASS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion fix

Operator ask: a small utility returning the active/needed work items to
dispatch from the roadmap (frontend/visualization seam, useful now).

- gunbc.roadmap_spawner gains roadmap_dispatch_json (single authority
  for spawn semantics): {ready (full node rows), upcoming (node +
  unmet_parents), done}. node_json also emits parent_node_ids (all
  parents — folds in claude's non-blocking #6110 finding; single
  parent_node_id kept for bridge back-compat). json_escape now escapes
  newlines (latent invalid-JSON bug, witnessed).
- dsl/tools/roadmap_dispatch.dag: `claim_batch --source-root dsl
  --entry dsl/tools/roadmap_dispatch.dag --function main --wet` writes
  target/roadmap-dispatch.json (schema roadmap-dispatch-file/v1, embeds
  the M0-M3 worker-brief template).
- First-receipt splits per operator dispatch review:
  1-resolver-pathology → -a (profile receipt only) / -b (fix one
  confirmed pathology, edge b←a); 2-live-read-runner-memory as the
  first slice of the read seam (seam gated on it); 2-privilege-model
  sized with RED/green acceptance.
- Edge-direction fix (operator structural concern): 2-converge-reland
  now DEPENDS ON 2-live-read-seam + 2-privilege-model (was inverted:
  privilege as converge's child); umbrella g2-runner edge removed so it
  cannot gate; periodic-actuation ← converge kept.

Verified by execution: spawner witnesses (new dispatch-partition +
newline-escape RED controls), authority witnesses, main_wet regen,
drift gate, doc-graph — PASS; utility run produces valid JSON with 11
ready / 4 gated, gates matching the signed ordering.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…owned

Operator go-ahead post-merge of #6110/#6111/#6112. 2-compile-clean-shard-a
(partition boundary + ONE shard + compose proof, no scheduler enrollment)
dispatched as adhoc-3e95c046-279 and marked owned; 2-compile-clean-shard-b
(floor-plan enrollment + before/after batch-1 wall receipt) gated on A.
Verified: regen, drift gate, authority + spawner witnesses PASS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
#6116 (gate-inventory audit + pilot red-control) merged without flipping
its node, unlike #6120 which self-marked. Set done: true to parity with
the resolver node — both now render "⏳ awaiting sign-off" (the [x]
checkbox is gated on operator sign-off by design, not merge alone).

Verified: main_wet regen, drift gate, roadmap_authority_witnesses PASS;
dispatch file moves 3-audit-gate-inventory to the done partition.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…phan

Addresses cursor REQUEST_CHANGES on #6122:
- Finding 3 (real): removed 1-resolver-pathology-receipt, a #6120-introduced
  near-verbatim duplicate of 1-resolver-pathology-a (§2 parallel representation).
- Findings 1/2 (drift-heal, not regression): done:true renders
  "[ ] — ⏳ awaiting sign-off" by design (box = signoff_accepts, not merge);
  #6120 had committed ROADMAP.md with resolver-A as [x] with no sign-off —
  that was drift, and the regen heals it to the honest awaiting-sign-off state.
- Also heals a pre-existing doc-graph orphan: mechanism-inventory-red-controls.md
  (#6116) was link-unreachable; attached it to the 3-audit-gate-inventory node
  via a new with_plan helper (keeps the node sized).

Verified: main_wet, drift gate, roadmap_authority_witnesses, doc-graph
orphans + dangling links all PASS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…branch

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…des done

The three verification-target PRs merged without self-marking their nodes;
central-marker flips 2-cd-transport, 2-live-read-runner-memory, and
2-compile-clean-shard-a to done: true (render 'awaiting sign-off' — operator
sign-off stays separate). Drift gate, roadmap witnesses, doc-graph all green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
briansrls and others added 12 commits July 2, 2026 04:54
…) done; flip 2-privilege-model (#6114); mark 2-emit-partition dispatched

- authored_merged_prs += 6104, 6106 (operator merged both)
- 2-privilege-model done: #6114 merged, live receipts on #6123/#6130 wet runs
- 2-emit-partition now un-gated and dispatched (adhoc-2040cdfe-46b)
- regen ROADMAP.md via generated_artifact_gate main_wet; drift gate + roadmap witnesses PASS

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ci_deploy_access to main)

The manual-deploy investigation surfaced two real bugs (dash/pipefail + principal
fiction) but the fix belongs in the modeled grounded-principal lane, not anemic
stopgap strings. Restoring these two files to origin/main clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lt stays modeled)

Operator directive 2026-07-02: a healthy floor is ~9min solo — a job past 30min is
pathological and must fail fast + release its runner slot instead of squatting toward
the platform ceiling (observed: 6h-timeout jobs holding all 25 fleet slots in a
self-sustaining queue deadlock, ~9h single-core claim_executor runs).

- gunbc_ci_job_timeout_policy_minutes: Int = 30 (+ Terminal disposition carrying the rationale)
- extdeps default_job_timeout_minutes = 360 deliberately unchanged (§3: it faithfully
  models GitHub's real platform default; policy lives in the workflow layer)
- ci.yml regenerated via generated_artifact_gate main_wet; drift gate PASS

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Thanks — noted on the Duration nit. Agreed a modeled Duration measure would be the more principled long-term shape; keeping it as a bare Int here matches the upstream field it feeds (GitHub's timeout-minutes) and the existing default_job_timeout_minutes: Int, so no §3 fork is introduced. If/when the measure/units tower grows a Duration, both this policy value and the upstream default migrate together as one change. No code change this PR. — sent from sunny-newt-884

briansrls and others added 2 commits July 2, 2026 16:07
…egression)

#6127 (merged 15:08 today) sharded compile-clean into ~1,605 per-module gunbc-compile
subprocesses, each re-resolving its full import closure (sum-of-closures ~30.9x the
monolith CPU), scheduled at WIDTH 1 because the width fold priced each single-module
shard against a stale whole-corpus 4.24GiB RSS sample vs the 8GiB cap. Live regression:
8h57m wall / 8h07m CPU per job on an 87%-idle 128-core box. Pre-regression monolith
green runs were 28-96min.

Sets the modeled rollback valve GUNBC_CI_COMPILE_CLEAN_WHOLE_TREE=1 on the ci job (env),
restoring the whole-tree monolith compile-clean. Authority = the matching v2.workflow
valve; rationale on ci_compile_clean_rollback_dissolution_trigger; Scaffold disposition
bound, dissolves with the v2 authority once the in-process shard resolve pool lands.

ci.yml regenerated via generated_artifact_gate main_wet; drift gate PASS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gunbai-bot gunbai-bot Bot changed the title CI policy: 30-minute job timeout (workflow-layer policy over the modeled GitHub default) CI emergency response: fail-fast 30min timeout + roll back #6127 shard-b regression Jul 2, 2026
@gunbai-bot

gunbai-bot Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Re-scoped to the full CI emergency response (adds a second commit — re-review expected).

Root cause found (not what we thought): the 8h57m jobs are not hardware contention — srv1 is 87% idle (128 cores, load 11, 45/125GB, 0 swap/OOM). They're an acute #6127 regression: shard-b sharded compile-clean into ~1,605 per-module gunbc compile subprocesses, each re-resolving its full import closure (~30.9× the monolith's CPU), scheduled at width 1 because the width fold priced each single-module shard against a stale whole-corpus 4.24 GiB RSS sample against the 8 GiB cap (real worst shard: 0.70 GiB).

This PR now does two things:

  1. gunbc_ci_job_timeout_policy_minutes = 30 (workflow-layer policy; GitHub's 360 default stays modeled) — the fail-fast smoke alarm.
  2. GUNBC_CI_COMPILE_CLEAN_WHOLE_TREE=1 on the ci job — rolls back the width-1 sharding to the whole-tree monolith.

Honest expectation: pre-regression monolith green runs were 28–96 min, so 30 min will still red most runs until the resolver single-thread pathology fix lands (separate lane, verified locally). This PR stops the fleet from burning 8-hour jobs and makes failure fast + honest; it does not by itself turn CI green. Receipts: ci.yml regenerated via generated_artifact_gate main_wet, drift gate PASS.

— sent from sunny-newt-884

briansrls and others added 2 commits July 2, 2026 16:25
The GUNBC_CI_COMPILE_CLEAN_WHOLE_TREE valve does NOT cleanly restore a green
monolith floor: under the valve, gunbc_ci_floor_schedule_lens_holds() FAILs and
batch-0 carries neither shards nor the monolith gate label (verified by execution
with claim_batch --wet under GUNBC_CI_COMPILE_CLEAN_WHOLE_TREE=1). The valve looks
like a half-exercised scaffold from #6127. Flipping it and patching witnesses to
match an unverified plan path would violate DESIGN §5 (ship only what runs green),
so this PR keeps only the sound, approved part:

- gunbc_ci_job_timeout_policy_minutes = 30 (workflow-layer; GitHub 360 default stays modeled)
- disposition text reconciled: 30min is deliberately BELOW today's 28-96min floor — a
  fail-fast smoke alarm until the resolver single-thread fix lands (verification local meanwhile)

The #6127 regression is better addressed by a clean revert of #6127 or a proper fix
to the plan's rollback branch (load-bearing floor-plan work) — surfaced to the operator.

ci.yml regenerated (timeout-only, no rollback env); drift gate PASS; both shard-b
floor witnesses PASS at origin/main state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gunbai-bot gunbai-bot Bot changed the title CI emergency response: fail-fast 30min timeout + roll back #6127 shard-b regression CI policy: 30-minute job timeout (workflow-layer policy over the modeled GitHub default) Jul 2, 2026
@gunbai-bot

gunbai-bot Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Backed the rollback valve-flip out of this PR — it now delivers only the 30-min timeout policy. @cursor your finding 1 was right, and pursuing it surfaced something worse.

Finding 1 confirmed and deeper. I made the two flagged witnesses rollback-aware, then verified by execution (claim_batch --wet under GUNBC_CI_COMPILE_CLEAN_WHOLE_TREE=1) — and the valve does not cleanly restore a green monolith floor:

  • compile_clean_sharding_enabled() → false ✓
  • witness_floor_batch_one_uses_compile_clean_shards() → false ✓
  • witness_floor_batch_one_has_no_compile_gate_monolith() → true (batch-0 carries no monolith gate label)
  • gunbc_ci_floor_schedule_lens_holds() → false

So under the valve, batch-0 has neither shards nor the monolith gate, and the schedule lens fails. The GUNBC_CI_COMPILE_CLEAN_WHOLE_TREE valve looks like a half-exercised scaffold from #6127. Patching witnesses to match an unverified plan path would violate §5 (ship only what runs green), so I reverted the valve-flip and the witness edits.

This PR now = the clean, approved change only: gunbc_ci_job_timeout_policy_minutes = 30 (workflow layer; extdeps 360 default untouched). Net diff is ci.yml (360→30 ×3) + ci_workflow.dag (policy data + disposition + sites). Drift gate PASS; both shard-b floor witnesses PASS at their original state. Disposition text reconciled with the 28–96min receipt @cursor flagged in finding 2.

Finding 2 addressed: the '~9min' text is gone; the disposition now states plainly that 30min is deliberately below today's 28–96min floor — a fail-fast smoke alarm until the resolver fix lands, verification local meanwhile.

The #6127 regression itself is now routed to the operator with a concrete recommendation (a clean git revert of #6127 is viable — 17 files, one trivial docs conflict — vs. properly fixing the plan's rollback branch).

— sent from sunny-newt-884

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant