Skip to content

Refresh stale CI-floor per_shard_peak (2.1→3.0 GiB, post-#5860) → spawn_width 3→2, fixes exit-137 OOM - #5875

Merged
briansrls merged 1 commit into
mainfrom
fix/ci-floor-pershard-refresh
Jun 27, 2026
Merged

briansrls merged 1 commit into
mainfrom
fix/ci-floor-pershard-refresh

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Jun 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

main is exit-137 again (run 28296782925, HEAD #5860) — not a logic regression. The memory-aware spawn_width formula trusted a stale committed per-shard constant.

gunbc_ci_floor_per_shard_peak_rss_bytes was 2.1 GiB (measured @ width=4, CI run 27893441091). The corpus grew (#5860 added witness files), pushing the real per-shard above ~2.67 GiB, so 3 concurrent shards at the formula-chosen width=3 exceeded the 8 GiB runner cap → claim_executor batch-2 OOM-killed. The formula was correct; its input lagged reality.

Fix

Re-measured the post-#5860 floor locally (main HEAD 6461042c64, pinned width=2): floor peak VmHWM = 6.0 GiB → per-shard = 6.0/2 = 3.0 GiB (3226056704). Refreshed the committed measured fact to that value at sample_width=2 — the operating width, where per-shard is conservatively highest (fewer/fatter shards) = fail-closed. The formula now derives width=2 (floor(0.8·8GiB / 3.0GiB) = 2), which fits at 6.0 GiB with ~2 GiB margin.

Also refreshed gunbc_ci_floor_measured_peak (the crude fallback sample) to the same width=2 run so the direct ≤ derived invariant holds; the strict < was an artifact of two historical runs and is corrected to ≤ (the direct measurement is a tight bound — equal when both derive from one unified measurement). Drift-monitor assertions updated to the new constants (sample_width 4→2).

This is the designed drift-monitor mechanism (a refreshed measured fact), not a cap-bump ratchet.

Verified by execution

  • gunbc_ci_floor_spawn_width_from_tree() = HardwareThreadCount { count: 2 } (was 3)
  • ci_floor_measurement_per_shard_rss_witnesses → true; all ci_floor_plan_witness_test test fns → true
  • Local floor at width=2: peak 6.0 GiB < 8 GiB cap

Durable follow-up

The committed scalar lags every corpus bump — this recurs. The real fix is already a roadmap item: 1-sched-resource-aware (docs/plans/resource-aware-scheduler.md) — auto-measure per-shard each run and derive width from it, retiring the static data row.

🤖 Generated with Claude Code

Invariant for the < → ≤ change (the one loosening edit — read before approving)

The drift monitor witness_direct_is_at_most_derived now asserts direct ≤ derived (was <). This is only sound because both operands are now drawn from the SAME measurement run (this PR sets gunbc_ci_floor_per_shard_rss_measurement and gunbc_ci_floor_measured_peak from one width=2 floor run): the direct per-shard and the crude total/concurrency fallback are then equal by construction, and ≤ asserts the direct measurement is a valid tight bound on the fallback — never larger. The original strict < only held because the two constants came from two different historical runs (8.4GiB/4 vs 11.6GiB/4); under that regime < was incidental, and once the corpus pushed the direct value past the stale fallback it flipped red — which is exactly the staleness this PR fixes. If a future edit ever sources direct and derived from different runs again, restore the strict < (equal-from-different-runs would be the suspicious case ≤ must not silently admit). The ≤ is not loosening drift detection — direct > derived (a refined number worse than the crude estimate) still fails it.

…dth 3->2, fixes exit-137

The memory-aware spawn_width formula trusted a stale committed per-shard
constant. gunbc_ci_floor_per_shard_peak_rss_bytes was 2.1 GiB (measured
@ width=4, CI run 27893441091); the corpus grew (#5860) so real per-shard
exceeded ~2.67 GiB and 3 concurrent shards at width=3 exceeded the 8 GiB
runner cap -> claim_executor batch-2 OOM-killed (exit 137, run 28296782925).

Re-measured post-#5860 floor locally (main HEAD 6461042, pinned width=2):
floor peak VmHWM = 6.0 GiB -> per-shard 6.0/2 = 3.0 GiB (3226056704).
Refreshed the committed measured fact to that value @ sample_width=2 (the
operating width; per-shard is conservatively higher at lower width =
fail-closed). Formula now derives width=2 (floor(0.8*8GiB/3.0GiB)=2), fits
at 6.0 GiB with ~2 GiB margin. Verified by execution:
gunbc_ci_floor_spawn_width_from_tree() = 2; per-shard + plan witnesses green.

Also refreshed gunbc_ci_floor_measured_peak (crude fallback sample) to the
same width=2 run so direct<=derived holds; the strict < was an artifact of
two historical runs, corrected to <= (direct is a tight bound, equal when
both derive from one unified measurement). Drift-monitor assertions updated.

Durable follow-up: roadmap item 1-sched-resource-aware
(resource-aware-scheduler.md) auto-measures per_shard each run so the scalar
no longer lags every corpus bump.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gunbai-bot gunbai-bot Bot changed the title Own and land the CI-floor OOM fix end-to-end: resolve-time per-shard RSS retention. Orphaned durable root is PR 5834 resolved-graph minimal-representation env-dedup on dead session tidy-wren-707; mitigations are PR 5847 width over-subscription and PR 5831 cgroup budget. Inherit cool-ram-182 memory-l Refresh stale CI-floor per_shard_peak (2.1→3.0 GiB, post-#5860) → spawn_width 3→2, fixes exit-137 OOM Jun 27, 2026
@gunbai-bot
gunbai-bot Bot marked this pull request as ready for review June 27, 2026 18:19
@briansrls
briansrls merged commit ef48361 into main Jun 27, 2026
2 of 4 checks passed
@briansrls
briansrls deleted the fix/ci-floor-pershard-refresh branch June 27, 2026 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant