Repository navigation
CI-floor OOM root-cause fix: within-run width fold over-subscribes the 8GiB runner cap (reserved=0 + reservation-not-cap budget + uniform-avg per-shard + stale measurement) -> claim_executor batch-2 OOM-killed (exit 137) fleet-wide - #5847
Closed
gunbai-bot[bot] wants to merge 25 commits into
Closed
gunbai-bot[bot] wants to merge 25 commits into
gunbai-bot[bot] wants to merge 25 commits into
Conversation
…g parser
The multi-line `let effective = \n if ...` form was rejected by the .dag parser
("expected expression, found Newline"). Restructure to call
gunbc_ci_floor_spawn_width_for_budget directly in each if branch, which also
avoids a Nat/Int type mismatch between the two branches.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
At spawn_width=2 the runner (memory.max=8GiB) OOM-killed batch-2 (CI run 28215907472). Lower bound on per-share: (8GiB - base~0.7GiB)/2 ~= 3.65GiB; set conservatively to 4GiB so floor(8GiB*4/5 / 4GiB) = 1 -> width=1. Disposition explicitly marks this as INFERRED (not directly sampled) with sample_width=2 recording the OOM-concurrency. Named dissolution trigger: width-sweep slope to replace the inferred bound with a real measurement. Adds witness_floor_spawn_width_at_cap_is_one as a discriminating assertion that the 8GiB cap produces exactly width=1. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…iscovery_rows_chunked The index itself always scans full source roots; peak reduction comes from dropping parse_cache/typed_module_cache between chunks, not partitioning the on-disk source index. Corrects misdescription flagged in review. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…musl cfg gate The previous approach rebuilt build_multi_entry_index (source_files + intern_table + empty caches) for every chunk. This was wasteful: source_files and intern_table are monotonic/content-addressed and safe to share across chunks. Only parse_cache and typed_module_cache (the heavy per-entry AST data) need clearing. Changes: - Add MultiEntryIndex::clear_entry_caches(): drops parse_cache and typed_module_cache while keeping intern_table and source_files alive across chunks - run_discovery_rows_chunked now builds the index once, runs each chunk against it, clears caches between chunks - heap_trim(): gate changed from cfg(target_os="linux") to cfg(all(target_os="linux", target_env="gnu")) — malloc_trim is glibc-only, absent on musl; the previous gate would break musl linking - Add RSS logging after each chunk via /proc/self/status so CI logs show memory trend - ci_floor_measurement.dag: change gunbc_ci_floor_per_shard_rss_source from Terminal to Scaffold (the Terminal's own prose named a dissolution trigger, contradicting the Terminal disposition — Scaffold is the honest tag for a pending measurement) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…easurement_mode_is_measured from test The per_shard_rss_measurement carried a *derived* value (whole-run cgroup peak / 4) under the PerShardRssPeakMeasured variant, which contradicts §3: the variant name claims direct measurement. Switch to PerShardRssPeakUnmeasured so gunbc_ci_floor_per_shard_peak_rss_bytes() uses the concurrent_memory_peak_per_share fallback (same computed value: 11592347648/4 = 2898086912). Remove witness_measurement_mode_is_measured() from the test conjunction — it returns false now (correct: we have no direct per-chunk measurement yet); the remaining witnesses still discriminate. Also remove stray double blank line in cli_run.rs left by removing the clear_entry_caches impl block. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rminal reason text Without JEMALLOC_SYS_WITH_MALLOC_CONF, jemalloc uses a 10-second dirty-page decay (dirty_decay_ms=10000), keeping freed pages in its arena RSS until the decay timer fires. This raised post-chunk RSS to 942-1242 MiB (vs ~114 MiB with glibc+malloc_trim) because the chunk completes before the 10s decay fires. Setting dirty_decay_ms:0,muzzy_decay_ms:0 makes jemalloc return dirty pages immediately via madvise(MADV_DONTNEED), giving the sawtooth baseline the fix requires. Embedded at build time via JEMALLOC_SYS_WITH_MALLOC_CONF in .cargo/config.toml (same pattern as RUST_MIN_STACK / RUST_TEST_THREADS). Also update the Terminal reason for gunbc_ci_floor_per_shard_rss_source to describe why the measurement is currently absent (chunking fix in flight, PerShardRssPeakUnmeasured fallback in use) rather than narrating the derivation that already lives in gunbc_ci_floor_per_shard_peak_rss_bytes(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…cumulation With spawn_width=2, both shards run in the same process; /proc/self/status VmRSS combines both shards. Later entry groups import much larger module closures, so typed_module_cache peaks non-linearly across a chunk. Factor=8 (25 groups/chunk vs 50) halves the worst-case unique-module accumulation, targeting combined peak < 8 GiB runner cgroup cap. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ined peak < 4 GiB Factor=8 (25 groups/chunk) still OOM-killed: complex entries (src/v2/workflow/) have large import closures, per-shard chunk peak reaching ~3780 MiB combined. Factor=16 (12-13 groups/chunk) halves unique-module accumulation again, targeting single-shard peak < 2 GiB → combined (2 shards) < 4 GiB < 8 GiB cgroup cap. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ctx.take() makes the drop-for-side-effect intent explicit and avoids the rustc unused_assignments lint (ctx = None is flagged because the None value is overwritten before being read; take() documents the drop). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…chunk-derived-per-share
PerShardRssPeakMeasured { 4294967296 } is now live (prev commit); the old
witness_per_shard_is_chunk_derived_per_share() checks the fallback value
2898086912 which no longer matches — replace with witness_measurement_mode_is_measured()
(now true since the data is in the Measured arm).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
briansrls
marked this pull request as ready for review
June 26, 2026 19:17
Contributor
Author
|
Superseded by jolly-carp-807 PR-A+B: honest .dag width fold with grounded reserved + max-observed per_shard + margin term. jemalloc/.cargo seed edit wrong layer per manager. |
3 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Auto-opened by session-dashboard for session
smart-lynx-515.Pushing to
session/smart-lynx-515advances this PR.Worker attestation
Before flipping this PR to ready for review, confirm each item:
npm test,cargo test) and the result.Closes #Ndirective.Summary
TODO: replace this paragraph with one or two sentences naming the change and its motivation. Reviewers read this first.
Test plan