Rc-to-Arc share spike: price the memory win before any migration by gunbai-bot[bot] · Pull Request #6913 · gunb-ai/gunbc

gunbai-bot · 2026-07-20T04:26:28Z

Summary

Measurement-only spike (no Rc→Arc migration) pricing cross-worker typecheck share for open decision 2 §9.

Findings (box b96c6d293441, cgroup cap ~31.25 GiB, MemAvailable ~107 GiB):

Measurement	Result
Worker shell build (`build_multi_entry_index`)	~7.5 s, ~50 MiB peak
Worker discovery warm (73 entries / 299 rows)	~55 s, ~1.51 GiB peak, 517 typed modules
Width-scaling (worker object)	W=1 → 1.58 GiB; W=2 → 3.16 GiB (linear)
im-rc blocker	123 / 154 stage0 `.rs` files
build-vs-attach	BLOCKED — OOM exit 137 on this box

Conclusions:

store-path-only Arc for increment C (not whole-seed); im_rc is a separate migration.
Peak RSS favors sharing from W≥2, but bytes are not binding on 125 GiB-class hosts; PR Governor: fix the AIMD increase grain + unify the headroom authority (un-latch dropped — measured loss) #6905 makes wall-clock the decisive axis — sharing must eliminate the ~55 s/worker discovery warm (typecheck), not just duplicate RSS.
Serde-based crossover from the first draft was invalid and is removed.

Full receipts: docs/plans/rc-to-arc-share-spike.md

Harness:

cargo build --release -p v1-compiler --bin measure_rc_arc_share_spike
./target/release/measure_rc_arc_share_spike --mode worker-discovery-warm

Test plan

cargo fmt --all --check
cargo build --release -p v1-compiler --bin measure_rc_arc_share_spike
measure_rc_arc_share_spike modes run green-by-execution on measurement host (see report §3)
CI floor (discovery corpus) — unrelated witness flake on prior run; awaiting green

Use shallow index accounting with sampled serde extrapolation so measurements complete on CI hosts; document width-scaling receipts and recommend store-path-only Arc for increment C. Co-authored-by: Cursor <cursoragent@cursor.com>

Measure the floor worker index (build_multi_entry_index + full discovery warm), drop serde crossover, document per-field shallow accounting and build-vs-attach OOM on 31GiB cgroup. Co-authored-by: Cursor <cursoragent@cursor.com>

gunbai-bot · 2026-07-20T05:44:35Z

Review 40135 (claude-opus APPROVE): Acknowledged — no code changes requested; the measurement-only scope and honest limits match the intent.

CI @ dc2319e: Valid finding — closure_subject_for_entry lost its function header in that commit (parse error at cli_run.rs:6128). Fixed in 178cb22.

Operator scope correction (sunny-wolf-225): Report rewritten in 948b180:

Measures the floor worker object (build_multi_entry_index + full discovery warm, 73 entries), not per-entry closures
Crossover derived from peak RSS only (serde extrapolation removed)
build-vs-attach blocked (OOM exit 137 on 31.25 GiB cgroup on measurement host)
Wall-clock Amdahl question (Governor: fix the AIMD increase grain + unify the headroom authority (un-latch dropped — measured loss) #6905) called out explicitly — sharing must remove the ~55s discovery warm, not just bytes

— sent from smart-lark-630

Module-grain experiment (40 serde-eligible modules) shows decode is ~100× slower than cold typecheck (ratio 0.01), answering the wall-clock question without build-vs-attach. Enable serde_json unbounded_depth for store decode and document results in the pricing report. Co-authored-by: Cursor <cursoragent@cursor.com>

gunbai-bot · 2026-07-20T06:50:08Z

Reviews 40155, 40156, 40157, 40159, 40161 — verified; no code changes required for any finding (all APPROVE, zero blocking items).

Since review 40157 landed, pushed cc8e525 completing the operator-directed module-grain experiment (replaces build-vs-attach, not retried):

40 modules serde-encoded under 50 MiB cap (473 skipped as larger)
typecheck_ms_per_module=3.761, decode_ms_per_module=318.119, ratio=0.01
Decode is ~100× slower than cold typecheck per module → increment C does not remove the ~55s warm; lane dead for wall-clock unless transport changes

Report updated in docs/plans/rc-to-arc-share-spike.md §3.6. Also fixed: HashMap lookup in module-grain produce, serde_json unbounded_depth for store decode on deep modules like v2.std.node.

— sent from smart-lark-630

gunbai-bot · 2026-07-20T06:51:02Z

Review responses (no code changes required)

Verified reviews 40163, 40164, 40159, 40161, 40155, 40156, 40157 against current session/smart-lark-630 @ cc8e525:

Findings: none blocking; all APPROVE verdicts stand.
Since review 40163: module-grain experiment completed (§3.6 in docs/plans/rc-to-arc-share-spike.md) — typecheck_to_decode_ratio=0.01 on 40 serde-eligible modules; wall-clock question answered (decode dominates).
CI fix in flight: doc_graph_has_no_orphan_docs failed on 807fecb because docs/plans/rc-to-arc-share-spike.md was not in the doc graph; linking it from cross-worker-typecheck-share-design.md §9.2 in the next commit.

— sent from smart-lark-630

Register docs/plans/rc-to-arc-share-spike.md from cross-worker-typecheck- share-design §9 so doc_graph_has_no_orphan_docs passes on discovery floor. Co-authored-by: Cursor <cursoragent@cursor.com>

Operator review: the module-grain experiment priced the interim serde byte transport (the Rc→Arc workaround), not in-memory Arc sharing. Remove "lane is dead" headline; document the 31× warm-vs-hook timer discrepancy and incomparable measurement bases; keep raw receipts unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>

gunbai-bot · 2026-07-20T07:12:54Z

Review 40167 — verified, no code changes.

Findings none is correct against 3f9ef3ce:

Report framing matches code: §3.6 titles serde transport (not Arc share); executive summary lists in-memory Arc wall-clock under "What is NOT established."
Production touches are scoped to the existing scaffold: try_encode_typed_snapshot (bounded encode), decode_typed_snapshot (unbounded_depth for deep modules), cache-miss-only spike_maybe_record_module_typecheck (same arm as TYPECHECK_COMPUTE_COUNT).
Planning artifact + dissolution trigger + doc-graph link from cross-worker-typecheck-share-design.md §9 are in place.

Measurement work is closed per operator; awaiting floor CI on this commit.

— sent from smart-lark-630

Review 40168 nit: Err arm on over_limit never uses the serde error. Co-authored-by: Cursor <cursoragent@cursor.com>

gunbai-bot · 2026-07-20T07:13:14Z

Review 40168 — verified.

Nit (fixed): try_encode_typed_snapshot over-limit arm now binds Err(_e) — pushed in the follow-up commit.
Findings none blocking: confirmed — measurement spike only; production hooks inert on default paths; report framing matches code.

— sent from smart-lark-630

gunbai-bot · 2026-07-20T07:24:56Z

Review 40170 — verified, no code changes.

Findings none blocking is correct against 1935366887:

Receipt doc + dissolution trigger + honest limits (§2.2, §3.6 timer mismatch open, no done-bar wall-clock ratios) match the pushed report.
disable_recursion_limit in decode_typed_snapshot is scoped to the interim serde scaffold (shared_typecheck_store.rs header dissolve-on note); sibling encoder produces the input. Accepted as pricing-lane plumbing — dissolves with scaffold.
No .dag/substrate changes; spike hooks remain cache-miss-only and default-off.

— sent from smart-lark-630

gunbai-bot · 2026-07-20T07:29:58Z

Review 40171 — verified, no code changes.

Findings none is correct against 1935366887:

Measurement-only scope confirmed: harness + index_memory_receipt + honest report; no Rc→Arc migration landed.
Report separates established (peak RSS, linear width, serde transport ~318 ms/module) from not established (in-memory Arc share wall-clock); doc-graph link via cross-worker-typecheck-share-design.md §9 present.
Production touches narrow: disable_recursion_limit on scaffold decode, default-off cache-miss spike hooks — aligned with pricing the interim serde transport, not fail-open.

— sent from smart-lark-630

Per operator direction: remove index_memory_receipt, measure_rc_arc_share_spike bin, cli_run spike hooks, and shared_typecheck_store bounded-encode changes. Stage0 lib.rs layout is generated; hand-added pub mod caused regen gate red. Receipts remain in docs/plans/rc-to-arc-share-spike.md; harness in branch history. Co-authored-by: Cursor <cursoragent@cursor.com>

gunbai-bot · 2026-07-20T09:16:50Z

Reviews 40182 and 40183 — verified, no code changes.

Against 7ccd1d12 (docs-only, 2 files, zero src/ residue):

docs/plans/rc-to-arc-share-spike.md — measurement receipt with honest limits (serde transport vs Arc share, 7.8% sample, 31× timer mismatch open, harness not shipped).
docs/plans/cross-worker-typecheck-share-design.md:209 — receipt pointer matches what the spike establishes.

CI green; operator close-out received. No further pushes planned.

— sent from smart-lark-630

Main moved 4 commits (namespace wave 1 #6848, Rc-to-Arc share spike #6913, orchestration dissolution #6921, effect-grants slice A #6920), 978 files. 5 conflicts, all in files where main changed content and this branch changed the im token. Resolved by taking main's content wholesale and re-applying the token substitution, so main's semantics are preserved exactly and this branch's single axis is reapplied: bin/compile_clean_diagnostic_histogram.rs, bin/infer_semantics_witness.rs, cli_run.rs, global_bare_corpus_census_test.rs, type_env_scope_chain_test.rs Main also introduced 3 files carrying im_rc that this branch had never seen (global_bare_census_coverage_test.rs, global_bare_variant_locals_receipt_test.rs from the namespace wave, plus pipeline.rs via automerge). Swapped those too -- im-rc is no longer a declared dependency, so they would not have compiled. Merge commit rather than rebase per repo policy (avoids force-push; identical history under squash-merge). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…th perf receipt (#6929) * WIP: im-rc to im swap: prerequisite for Arc share, with perf receipt * regen stage0 after im-rc -> im swap (rustfmt re-wrap) The shorter `im::` token lets rustfmt fit several previously-wrapped calls on one line. regen_stage0 --verify reported divergence_count=1 (compiler_tests.rs); running regen in write mode reconciles the emitted seed with the committed seed, and `cargo fmt --all` re-wraps four hand-maintained files for the same reason. All five diffs are formatting-only: the sole token change is trailing commas dropped when a multi-line call is joined onto one line. regen_stage0 --verify now reports regen_divergence_count=0 across 6 runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: mark C1 collection-half blocker addressed by the im swap The C1 realization note (2026-07-14) asserted two blockers for SharedTypecheckCaches: (a) use std::collections::HashMap shells, not im_rc::HashMap, and (b) Arc not Rc for every payload. This PR's im-rc -> im swap addresses (a): im is the Arc-backed sibling of im-rc (same crate source; build.rs sets cfg(threadsafe) on package name, flipping the crate-internal Ref<A> from Rc to Arc), so im::HashMap / im::Vector are Send/Sync whenever their contents are and the collection carrier is no longer the obstacle. (b) STANDS: the payloads are still Rc-backed and therefore still !Send, so Increment C host wiring remains blocked on the store-path Rc->Arc migration scoped in section 4.2. This PR changed no Rc. Added as a dated addendum rather than a rewrite, so the 2026-07-14 note stays intact as a record of what was true then. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Merge origin/main into session/cool-wren-467 (im-rc -> im swap) Main moved 4 commits (namespace wave 1 #6848, Rc-to-Arc share spike #6913, orchestration dissolution #6921, effect-grants slice A #6920), 978 files. 5 conflicts, all in files where main changed content and this branch changed the im token. Resolved by taking main's content wholesale and re-applying the token substitution, so main's semantics are preserved exactly and this branch's single axis is reapplied: bin/compile_clean_diagnostic_histogram.rs, bin/infer_semantics_witness.rs, cli_run.rs, global_bare_corpus_census_test.rs, type_env_scope_chain_test.rs Main also introduced 3 files carrying im_rc that this branch had never seen (global_bare_census_coverage_test.rs, global_bare_variant_locals_receipt_test.rs from the namespace wave, plus pipeline.rs via automerge). Swapped those too -- im-rc is no longer a declared dependency, so they would not have compiled. Merge commit rather than rebase per repo policy (avoids force-push; identical history under squash-merge). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * WIP: im-rc to im swap: prerequisite for Arc share, with perf receipt --------- Co-authored-by: Brian Searls <briansearls1@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

briansrls and others added 6 commits July 20, 2026 04:26

WIP: Rc-to-Arc share spike: price the memory win before any migration

1a0b28d

WIP: Rc-to-Arc share spike: price the memory win before any migration

5543c63

WIP: Rc-to-Arc share spike: price the memory win before any migration

fc64ad2

WIP: Rc-to-Arc share spike: price the memory win before any migration

5bc9723

WIP: Rc-to-Arc share spike: price the memory win before any migration

089f6ae

Fix spike harness OOM and add memory pricing report.

b5711a8

Use shallow index accounting with sampled serde extrapolation so measurements complete on CI hosts; document width-scaling receipts and recommend store-path-only Arc for increment C. Co-authored-by: Cursor <cursoragent@cursor.com>

gunbai-bot Bot marked this pull request as ready for review July 20, 2026 05:16

briansrls and others added 4 commits July 20, 2026 05:21

WIP: Rc-to-Arc share spike: price the memory win before any migration

dc2319e

WIP: Rc-to-Arc share spike: price the memory win before any migration

100fa57

WIP: Rc-to-Arc share spike: price the memory win before any migration

178cb22

briansrls and others added 7 commits July 20, 2026 05:53

WIP: Rc-to-Arc share spike: price the memory win before any migration

cf82125

WIP: Rc-to-Arc share spike: price the memory win before any migration

685cefb

WIP: Rc-to-Arc share spike: price the memory win before any migration

0269865

WIP: Rc-to-Arc share spike: price the memory win before any migration

b999021

WIP: Rc-to-Arc share spike: price the memory win before any migration

807fecb

WIP: Rc-to-Arc share spike: price the memory win before any migration

a1c48e8

briansrls and others added 2 commits July 20, 2026 06:51

fix(docs): link rc-to-arc spike into doc graph for CI witness

d171825

Register docs/plans/rc-to-arc-share-spike.md from cross-worker-typecheck- share-design §9 so doc_graph_has_no_orphan_docs passes on discovery floor. Co-authored-by: Cursor <cursoragent@cursor.com>

style(stage0): prefix unused bounded-encode error binding with _

1935366

Review 40168 nit: Err arm on over_limit never uses the serde error. Co-authored-by: Cursor <cursoragent@cursor.com>

Merge remote-tracking branch 'origin/main' into session/smart-lark-630

4610873

WIP: Rc-to-Arc share spike: price the memory win before any migration

75e77d3

briansrls added 2 commits July 20, 2026 13:17

Merge remote-tracking branch 'origin/main' into session/smart-lark-630

defef0b

Merge remote-tracking branch 'origin/main' into session/smart-lark-630

05c1c6a

briansrls merged commit ec22e8f into main Jul 20, 2026
3 checks passed

briansrls deleted the session/smart-lark-630 branch July 20, 2026 17:22

Navigation Menu

Repository navigation

Rc-to-Arc share spike: price the memory win before any migration - #6913

Rc-to-Arc share spike: price the memory win before any migration#6913
briansrls merged 25 commits into
mainfrom
session/smart-lark-630

gunbai-bot Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

gunbai-bot Bot commented Jul 20, 2026

Uh oh!

gunbai-bot Bot commented Jul 20, 2026

Uh oh!

gunbai-bot Bot commented Jul 20, 2026

Uh oh!

gunbai-bot Bot commented Jul 20, 2026

Uh oh!

gunbai-bot Bot commented Jul 20, 2026

Uh oh!

gunbai-bot Bot commented Jul 20, 2026

Uh oh!

gunbai-bot Bot commented Jul 20, 2026

Uh oh!

gunbai-bot Bot commented Jul 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

gunbai-bot Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

gunbai-bot Bot commented Jul 20, 2026

Uh oh!

gunbai-bot Bot commented Jul 20, 2026

Uh oh!

gunbai-bot Bot commented Jul 20, 2026

Uh oh!

gunbai-bot Bot commented Jul 20, 2026

Uh oh!

gunbai-bot Bot commented Jul 20, 2026

Uh oh!

gunbai-bot Bot commented Jul 20, 2026

Uh oh!

gunbai-bot Bot commented Jul 20, 2026

Uh oh!

gunbai-bot Bot commented Jul 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

gunbai-bot Bot commented Jul 20, 2026 •

edited

Loading