Repository navigation
Refresh stale CI-floor per_shard_peak (2.1→3.0 GiB, post-#5860) → spawn_width 3→2, fixes exit-137 OOM - #5875
Merged
Merged
Conversation
…dth 3->2, fixes exit-137 The memory-aware spawn_width formula trusted a stale committed per-shard constant. gunbc_ci_floor_per_shard_peak_rss_bytes was 2.1 GiB (measured @ width=4, CI run 27893441091); the corpus grew (#5860) so real per-shard exceeded ~2.67 GiB and 3 concurrent shards at width=3 exceeded the 8 GiB runner cap -> claim_executor batch-2 OOM-killed (exit 137, run 28296782925). Re-measured post-#5860 floor locally (main HEAD 6461042, pinned width=2): floor peak VmHWM = 6.0 GiB -> per-shard 6.0/2 = 3.0 GiB (3226056704). Refreshed the committed measured fact to that value @ sample_width=2 (the operating width; per-shard is conservatively higher at lower width = fail-closed). Formula now derives width=2 (floor(0.8*8GiB/3.0GiB)=2), fits at 6.0 GiB with ~2 GiB margin. Verified by execution: gunbc_ci_floor_spawn_width_from_tree() = 2; per-shard + plan witnesses green. Also refreshed gunbc_ci_floor_measured_peak (crude fallback sample) to the same width=2 run so direct<=derived holds; the strict < was an artifact of two historical runs, corrected to <= (direct is a tight bound, equal when both derive from one unified measurement). Drift-monitor assertions updated. Durable follow-up: roadmap item 1-sched-resource-aware (resource-aware-scheduler.md) auto-measures per_shard each run so the scalar no longer lags every corpus bump. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
main is exit-137 again (run 28296782925, HEAD #5860) — not a logic regression. The memory-aware
spawn_widthformula trusted a stale committed per-shard constant.gunbc_ci_floor_per_shard_peak_rss_byteswas 2.1 GiB (measured @ width=4, CI run 27893441091). The corpus grew (#5860 added witness files), pushing the real per-shard above ~2.67 GiB, so 3 concurrent shards at the formula-chosen width=3 exceeded the 8 GiB runner cap →claim_executorbatch-2 OOM-killed. The formula was correct; its input lagged reality.Fix
Re-measured the post-#5860 floor locally (main HEAD
6461042c64, pinned width=2): floor peak VmHWM = 6.0 GiB → per-shard = 6.0/2 = 3.0 GiB (3226056704). Refreshed the committed measured fact to that value atsample_width=2— the operating width, where per-shard is conservatively highest (fewer/fatter shards) = fail-closed. The formula now derives width=2 (floor(0.8·8GiB / 3.0GiB) = 2), which fits at 6.0 GiB with ~2 GiB margin.Also refreshed
gunbc_ci_floor_measured_peak(the crude fallback sample) to the same width=2 run so thedirect ≤ derivedinvariant holds; the strict<was an artifact of two historical runs and is corrected to≤(the direct measurement is a tight bound — equal when both derive from one unified measurement). Drift-monitor assertions updated to the new constants (sample_width4→2).This is the designed drift-monitor mechanism (a refreshed measured fact), not a cap-bump ratchet.
Verified by execution
gunbc_ci_floor_spawn_width_from_tree()=HardwareThreadCount { count: 2 }(was 3)ci_floor_measurement_per_shard_rss_witnesses→ true; allci_floor_plan_witness_testtest fns → trueDurable follow-up
The committed scalar lags every corpus bump — this recurs. The real fix is already a roadmap item:
1-sched-resource-aware(docs/plans/resource-aware-scheduler.md) — auto-measure per-shard each run and derive width from it, retiring the static data row.🤖 Generated with Claude Code
Invariant for the
<→≤change (the one loosening edit — read before approving)The drift monitor
witness_direct_is_at_most_derivednow assertsdirect ≤ derived(was<). This is only sound because both operands are now drawn from the SAME measurement run (this PR setsgunbc_ci_floor_per_shard_rss_measurementandgunbc_ci_floor_measured_peakfrom one width=2 floor run): the direct per-shard and the crudetotal/concurrencyfallback are then equal by construction, and≤asserts the direct measurement is a valid tight bound on the fallback — never larger. The original strict<only held because the two constants came from two different historical runs (8.4GiB/4 vs 11.6GiB/4); under that regime<was incidental, and once the corpus pushed the direct value past the stale fallback it flipped red — which is exactly the staleness this PR fixes. If a future edit ever sourcesdirectandderivedfrom different runs again, restore the strict<(equal-from-different-runs would be the suspicious case≤must not silently admit). The≤is not loosening drift detection —direct > derived(a refined number worse than the crude estimate) still fails it.