Skip to content

interp: phase-0 memory measurement harness (ctrl#1533) - #4653

Merged
briansrls merged 11 commits into
mainfrom
session/neat-ibex-297
Jun 11, 2026
Merged

briansrls merged 11 commits into
mainfrom
session/neat-ibex-297

Conversation

@briansrls

@briansrls briansrls commented Jun 11, 2026 •

Copy link
Copy Markdown
Contributor

Phase 0 of the persistent-structures plan (ctrl #1533, merged; v4 addendum ctrl #1534, merged): the measurement harness that turns the 5–10× estimate into ground truth and becomes the before/after receipt for the carrier swap. Read-only tooling — no semantic change to any evaluation path.

What it measures

  1. Copy-on-update work — MutationCounters on InterpContext (context-scoped, same discipline as interp: scope data cache + pure-call memo to InterpContext (drop process-global caches) #4644's caches): calls + entries copied for map_insert/merge/list_push/concat/set ops. Fold-building an n-entry collection shows the triangular term (copied ≈ n(n−1)/2); pinned by test (4 inserts → 6 entries copied).
  2. FreeMonoid flattening — counters on the free_monoid_to_vec chokepoint (ctrl#1476 B1). Thread-local, not context-scoped, because the chokepoint fires inside Value::eq (impl PartialEq, no ctx) — and unlike the caches interp: scope data cache + pure-call memo to InterpContext (drop process-global caches) #4644 scoped these are two fixed-size integers: no keys, no keepalive, no growth.
  3. Retained-value byte accounting — InterpContext::account_retained_memory: sharing-aware walk (visited-set on Rc pointers) over data cache + pure-call memo + extra roots; per-variant occurrence/unique/shared/bytes. Shared structure accounted exactly once (pinned by test).

Surface: GUNBC_INTERP_STATS=1 claim_batch ... prints the report on stderr (plus VmHWM/VmRSS).

Receipt on the real v4 gate workload

7 affected_testgen_*_holds witnesses (the node-frontier gate set), all PASS, 255 MB RSS:

  • Native mutation primitives: 0 calls. v4 collections are closure-chain maps + FreeMonoid trees (persistent by construction at the model level); the host copy cost rides the flatten chokepoint instead — exactly the L1/L2 layer split ctrl chore(r3-debt-paydown): velocity-tripwire script + first 2026-05-02 measurement (FIRED 25.33:1) #1534 documents. The carrier-swap phase must unify the FreeMonoid representation (the B1 'unified runtime representation' dissolve target), not just swap HashMap→HAMT.
  • Retained: 21.8 MB / 219K allocations. Record: 12.7 MB with 48K sharing hits; 31.8K un-interned Strings (1.06 MB) — the M-D interning/positional-record baseline.

Tests

Per ctrl #1533 this phase has no gate ('read-only tooling'); phase-1 modeling PRs (M-A/M-C at the v4.std.runtime authority per ctrl #1534) come next, separately.

🤖 Generated with Claude Code

Brian Searls and others added 8 commits June 10, 2026 23:35
Context-scoped MutationCounters on the copy-on-update primitives
(map_insert/merge/list_push/concat/set ops: calls + entries copied — the
triangular/quadratic receipt), thread-local flatten counters on the
free_monoid_to_vec chokepoint (fires inside Value::eq, so no ctx; two
fixed-size integers, not a cache), and a sharing-aware retained-value
byte accounting walk (per-variant counts/bytes, visited-set dedup) on
InterpContext. claim_batch prints the report under GUNBC_INTERP_STATS=1.

Receipt on the real v4 gate workload (7 affected_testgen witnesses, all
PASS): native mutation primitives 0 calls — v4 collections are closure
chains + FreeMonoid trees, so the host copy cost rides the flatten
chokepoint, confirming ctrl PR #1534's layer analysis. Retained: 21.8 MB
/ 219K allocations (Record 12.7MB with 48K sharing hits; 31K un-interned
Strings) — the M-D interning/positional-record receipt baseline.

Read-only tooling per ctrl#1533 phase 0; no semantic change to any
evaluation path (counters + opt-in report only).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@briansrls briansrls changed the title memory usage interp: phase-0 memory measurement harness (ctrl#1533) Jun 11, 2026
@briansrls
briansrls marked this pull request as ready for review June 11, 2026 01:47
Brian Searls and others added 2 commits June 11, 2026 02:07
…latten in claim_batch

Review #28288 (cursor/composer-2.5) on #4653: the copy-work counters mixed
definitions across dispatch paths, breaking the single-receipt property (P2).

- One definition per counter, by operation semantics, not dispatch path:
  add-one ops (map_insert/list_push/set_insert) count the receiver's
  pre-existing entries; merge ops (map_merge/list_concat/set_union) count
  both operands' entries. Method .concat/.append/.push buckets by what the
  arg IS (collection -> concat, atomic -> push); binop + and builtins agree.
- set_union gets its own row (was folded into set_insert — same P2 class).
- builtin map_merge instrumented (was counted on the method path only).
- claim_batch fm_flatten row is now a delta sampled across the witness loop,
  matching the context-scoped counters next to it.

Tests pin the unified semantics: concat counts both operands (9 for the
3⊕1 then 4⊕1 chain), atomic append lands in list_push with receiver-only
copy-work, and neither bucket leaks into the other.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@briansrls

Copy link
Copy Markdown
Contributor Author

Re review #28288 (cursor/composer-2.5, REQUEST_CHANGES): all three findings were valid — fixed in a44739f.

  1. Inconsistent list_concat_items_copied: counters now carry one definition per primitive, chosen by operation semantics rather than dispatch path (documented on MutationCounters): add-one ops (map_insert/list_push/set_insert) count the receiver's pre-existing entries; merge ops (map_merge/list_concat/set_union) count both operands' entries. The method/binop/builtin concat paths all agree now.
  2. push vs concat bucket split by dispatch path: a .concat/.append/.push method call buckets by what its argument IS — a flattening collection arg merges (list_concat), an atomic arg appends one element (list_push) — so the same primitive always lands in the same bucket regardless of surface. New tests pin both directions (no leakage either way).
  3. fm_flatten scope mismatch in claim_batch: the bin now samples a baseline after resolve/context setup and prints the delta over the witness loop, matching the context-scoped counters next to it (the row is labeled as a delta).

Also fixed two more instances of the same P2 class the review didn't list: set_union was incrementing the set_insert bucket (now its own row), and builtin map_merge was un-instrumented while the method path counted (now both count).

Receipt re-run on the 7-witness v4 gate workload: all PASS, numbers unchanged (fm_flatten 185 calls / 770 items — the delta confirms all flatten traffic is inside the witness loop), retained 21.8 MB / 219K allocations, RSS 255 MB. 7/7 tests green, clippy/fmt clean.

— sent from neat-ibex-297

@briansrls

Copy link
Copy Markdown
Contributor Author

Re review #28296 (claude/opus-4-7, APPROVE) — on the non-blocking note about account_value adding type_name.len() before the first-visit check in account_named_fields: verified against the code, and the per-occurrence accounting is intentional and correct, not a double-count. type_name is an owned String on each Value::Record/Value::Variant instance — only fields sits behind the shared Rc<HashMap>. Two Record occurrences sharing one fields map each own a distinct type_name heap allocation (cloning a Record clones the String), so counting those bytes per occurrence counts real, separate allocations. The first-visit guard correctly covers only the genuinely shared part (the fields map). No change needed.

— sent from neat-ibex-297

@briansrls
briansrls merged commit c1cdc9b into main Jun 11, 2026
11 checks passed
@briansrls
briansrls deleted the session/neat-ibex-297 branch June 11, 2026 03:59
briansrls added a commit that referenced this pull request Jun 11, 2026
…ntity laws (M-C) with executable witnesses (#4660)

* WIP: memory usage

* WIP: memory usage

* WIP: memory usage

* WIP: memory usage

* WIP: memory usage

* WIP: memory usage

* feat(interp): phase-0 memory measurement harness (ctrl#1533)

Context-scoped MutationCounters on the copy-on-update primitives
(map_insert/merge/list_push/concat/set ops: calls + entries copied — the
triangular/quadratic receipt), thread-local flatten counters on the
free_monoid_to_vec chokepoint (fires inside Value::eq, so no ctx; two
fixed-size integers, not a cache), and a sharing-aware retained-value
byte accounting walk (per-variant counts/bytes, visited-set dedup) on
InterpContext. claim_batch prints the report under GUNBC_INTERP_STATS=1.

Receipt on the real v4 gate workload (7 affected_testgen witnesses, all
PASS): native mutation primitives 0 calls — v4 collections are closure
chains + FreeMonoid trees, so the host copy cost rides the flatten
chokepoint, confirming ctrl PR #1534's layer analysis. Retained: 21.8 MB
/ 219K allocations (Record 12.7MB with 48K sharing hits; 31K un-interned
Strings) — the M-D interning/positional-record receipt baseline.

Read-only tooling per ctrl#1533 phase 0; no semantic change to any
evaluation path (counters + opt-in report only).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* WIP: memory usage

* fix(interp): one copy-work definition per primitive; delta-scope fm_flatten in claim_batch

Review #28288 (cursor/composer-2.5) on #4653: the copy-work counters mixed
definitions across dispatch paths, breaking the single-receipt property (P2).

- One definition per counter, by operation semantics, not dispatch path:
  add-one ops (map_insert/list_push/set_insert) count the receiver's
  pre-existing entries; merge ops (map_merge/list_concat/set_union) count
  both operands' entries. Method .concat/.append/.push buckets by what the
  arg IS (collection -> concat, atomic -> push); binop + and builtins agree.
- set_union gets its own row (was folded into set_insert — same P2 class).
- builtin map_merge instrumented (was counted on the method path only).
- claim_batch fm_flatten row is now a delta sampled across the witness loop,
  matching the context-scoped counters next to it.

Tests pin the unified semantics: concat counts both operands (9 for the
3⊕1 then 4⊕1 chain), atomic append lands in list_push with receiver-only
copy-work, and neither bucket leaks into the other.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* WIP: memory usage

* WIP: memory usage

* review: explicit collection imports + free-function concat in value_carrier_laws; strengthen insert-order witness to 3 keys

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Brian Searls <briansearls1@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
briansrls added a commit that referenced this pull request Jun 11, 2026
…iveContract + M-D name-interning declarations (#4666)

* WIP: memory usage

* WIP: memory usage

* WIP: memory usage

* WIP: memory usage

* WIP: memory usage

* WIP: memory usage

* feat(interp): phase-0 memory measurement harness (ctrl#1533)

Context-scoped MutationCounters on the copy-on-update primitives
(map_insert/merge/list_push/concat/set ops: calls + entries copied — the
triangular/quadratic receipt), thread-local flatten counters on the
free_monoid_to_vec chokepoint (fires inside Value::eq, so no ctx; two
fixed-size integers, not a cache), and a sharing-aware retained-value
byte accounting walk (per-variant counts/bytes, visited-set dedup) on
InterpContext. claim_batch prints the report under GUNBC_INTERP_STATS=1.

Receipt on the real v4 gate workload (7 affected_testgen witnesses, all
PASS): native mutation primitives 0 calls — v4 collections are closure
chains + FreeMonoid trees, so the host copy cost rides the flatten
chokepoint, confirming ctrl PR #1534's layer analysis. Retained: 21.8 MB
/ 219K allocations (Record 12.7MB with 48K sharing hits; 31K un-interned
Strings) — the M-D interning/positional-record receipt baseline.

Read-only tooling per ctrl#1533 phase 0; no semantic change to any
evaluation path (counters + opt-in report only).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* WIP: memory usage

* fix(interp): one copy-work definition per primitive; delta-scope fm_flatten in claim_batch

Review #28288 (cursor/composer-2.5) on #4653: the copy-work counters mixed
definitions across dispatch paths, breaking the single-receipt property (P2).

- One definition per counter, by operation semantics, not dispatch path:
  add-one ops (map_insert/list_push/set_insert) count the receiver's
  pre-existing entries; merge ops (map_merge/list_concat/set_union) count
  both operands' entries. Method .concat/.append/.push buckets by what the
  arg IS (collection -> concat, atomic -> push); binop + and builtins agree.
- set_union gets its own row (was folded into set_insert — same P2 class).
- builtin map_merge instrumented (was counted on the method path only).
- claim_batch fm_flatten row is now a delta sampled across the witness loop,
  matching the context-scoped counters next to it.

Tests pin the unified semantics: concat counts both operands (9 for the
3⊕1 then 4⊕1 chain), atomic append lands in list_push with receiver-only
copy-work, and neither bucket leaks into the other.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* WIP: memory usage

* WIP: memory usage

* review: explicit collection imports + free-function concat in value_carrier_laws; strengthen insert-order witness to 3 keys

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* model: ctrl#1533 phase-1 follow-up — M-B carrier-cost facts on PrimitiveContract + M-D name-interning declarations

M-B: PrimitiveContract gains carrier_cost: CarrierCostSensitivity. 12
carrier-sensitive primitives declare both arms (ephemeral = copy-before-
update, what the v2 interpreter does today; persistent = the declared M-A
carrier cost); 49 carrier-insensitive primitives marked explicitly.

M-D: name-interning facts at the v4.std.value_carrier authority — type/
variant/field names are references into the resolved graph's declared name
set; positional-record layout derives from this fact (phase 3 implements).

New witness name_interning_covers_all_domains_holds (8/8 green).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Brian Searls <briansearls1@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
briansrls added a commit that referenced this pull request Jun 11, 2026
…B (persistent updates, copied-counters collapse to 0) (#4672)

* WIP: memory usage

* WIP: memory usage

* WIP: memory usage

* WIP: memory usage

* WIP: memory usage

* WIP: memory usage

* feat(interp): phase-0 memory measurement harness (ctrl#1533)

Context-scoped MutationCounters on the copy-on-update primitives
(map_insert/merge/list_push/concat/set ops: calls + entries copied — the
triangular/quadratic receipt), thread-local flatten counters on the
free_monoid_to_vec chokepoint (fires inside Value::eq, so no ctx; two
fixed-size integers, not a cache), and a sharing-aware retained-value
byte accounting walk (per-variant counts/bytes, visited-set dedup) on
InterpContext. claim_batch prints the report under GUNBC_INTERP_STATS=1.

Receipt on the real v4 gate workload (7 affected_testgen witnesses, all
PASS): native mutation primitives 0 calls — v4 collections are closure
chains + FreeMonoid trees, so the host copy cost rides the flatten
chokepoint, confirming ctrl PR #1534's layer analysis. Retained: 21.8 MB
/ 219K allocations (Record 12.7MB with 48K sharing hits; 31K un-interned
Strings) — the M-D interning/positional-record receipt baseline.

Read-only tooling per ctrl#1533 phase 0; no semantic change to any
evaluation path (counters + opt-in report only).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* WIP: memory usage

* fix(interp): one copy-work definition per primitive; delta-scope fm_flatten in claim_batch

Review #28288 (cursor/composer-2.5) on #4653: the copy-work counters mixed
definitions across dispatch paths, breaking the single-receipt property (P2).

- One definition per counter, by operation semantics, not dispatch path:
  add-one ops (map_insert/list_push/set_insert) count the receiver's
  pre-existing entries; merge ops (map_merge/list_concat/set_union) count
  both operands' entries. Method .concat/.append/.push buckets by what the
  arg IS (collection -> concat, atomic -> push); binop + and builtins agree.
- set_union gets its own row (was folded into set_insert — same P2 class).
- builtin map_merge instrumented (was counted on the method path only).
- claim_batch fm_flatten row is now a delta sampled across the witness loop,
  matching the context-scoped counters next to it.

Tests pin the unified semantics: concat counts both operands (9 for the
3⊕1 then 4⊕1 chain), atomic append lands in list_push with receiver-only
copy-work, and neither bucket leaks into the other.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* WIP: memory usage

* WIP: memory usage

* review: explicit collection imports + free-function concat in value_carrier_laws; strengthen insert-order witness to 3 keys

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* model: ctrl#1533 phase-1 follow-up — M-B carrier-cost facts on PrimitiveContract + M-D name-interning declarations

M-B: PrimitiveContract gains carrier_cost: CarrierCostSensitivity. 12
carrier-sensitive primitives declare both arms (ephemeral = copy-before-
update, what the v2 interpreter does today; persistent = the declared M-A
carrier cost); 49 carrier-insensitive primitives marked explicitly.

M-D: name-interning facts at the v4.std.value_carrier authority — type/
variant/field names are references into the resolved graph's declared name
set; positional-record layout derives from this fact (phase 3 implements).

New witness name_interning_covers_all_domains_holds (8/8 green).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ctrl#1533 phase 2: swap Value::Map/List host carriers to im-rc HAMT/RRB

Value::Map now carries Rc<im_rc::HashMap> (HAMT, matches map_carrier
^hash_array_mapped_trie) and Value::List carries Rc<im_rc::Vector> (RRB,
matches list_carrier ^relaxed_radix_balanced_tree). Rc wrapper kept:
value_rc_identity memo keys and accounting dedup rely on Rc::as_ptr.

map insert/merge and list push/concat no longer copy entries; the
*_entries_copied / *_items_copied counters now count only FreeMonoid
chain flattens — the copied-counter collapse is the phase-2 receipt.
List ops route native List and Empty/Cons chains through one
value_to_list_carrier bridge (B1 alias transparency preserved).

New witnesses: list/map prior-version persistence in
value_carrier_laws.dag; host-side at-scale probes (n=500, past inline
chunks) in value_carrier_swap_test.rs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix at-scale probes: use DSL fold(list, init:, f: fn(){}) syntax

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Brian Searls <briansearls1@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant