perf(pmt): chunked-leaf packing for basic lists and container_struct by GrapeBaBa · Pull Request #346 · ChainSafe/lodestar-z

GrapeBaBa · 2026-05-05T08:07:30Z

Motivation

State transition is dominated by PMT operations on BeaconState's large basic-element lists (Balances, EpochParticipation, InactivityScores, ~1.4M items each) and per-field tree access for struct-shaped containers (Validator). On a mainnet fulu state with 2.18M validators, these account for the bulk of processEpoch and processBlock runtime.

Description

Five PMT-level changes that compose:

u64 payload column — collapses every node kind's payload (branch left/right Ids, chunked_leaf pointer, container_struct vtable pointer, free-list link) into one machine word. State packs [free_bit:1 | kind:3 | ref_count:28] in u32. Cache validity moves out of kind into a 0xFF… sentinel in the root column. Hot tree-walk visits touch only state (1 B) + 4-8 B from payload, fitting one cache line.
chunked_leaf — opt-in via opts.chunked_leaf=true on FixedListType / FixedVectorType. Bottom k_log2 = 6 levels of the chunks subtree fold into one *ChunkedLeaf heap blob holding K=64 chunks. For 1M-item List<u64> pool metadata drops ~64× (256K Node.Id → 4096 ChunkedLeaves + 4096 heap blobs). Bulk read/write get SIMD-batched root recomputation and amortized CoW (one 2 KB memcpy per dirty leaf instead of 6 path clones per dirty chunk).
container_struct (originally feat: model phase0 Validator as struct #232) — a node kind whose payload is *ContainerStructRef (vtable + caller-allocated T). Backs StructContainerType for Validator etc. Field access = O(1) struct read instead of per-field tree walk; hashTreeRoot calls type's cached get_root directly.
Pool dual-allocator — Pool keeps two allocators routed by allocation kind: page_allocator for the MultiArrayList node columns (one large, infrequent allocation), and allocator (default c_allocator) for every per-node out-of-line heap blob — ContainerStructRef, WrappedT, and the 2 KB chunked_leaf blobs. Page-per-alloc on the small lane wastes ~70 GB of virtual address space at 2.18M validators on macOS arm64 and thrashes the TLB; the bucket allocator packs them densely. Pool.init switched to options-struct shape (Pool.init(.{}) for production defaults). This unblocks serializeValidators / getEffectiveBalanceIncrementsZeroInactive / getSingleProof binding tests at mainnet scale (24 s → ~500 ms, 50× speedup, equal to main).
Zero-copy validator access — completes the container_struct value chain. PR feat: model phase0 Validator as struct #232 added pool.getStructPtr(node, T) but no list-iteration API was built on top, so callers still cloned the full 263 MB validators slice per epoch transition. This PR adds:
- StructContainerType.tree.getValuePtr(node, pool) -> *const T — direct typed pointer into the pool's container_struct payload.
- ListCompositeTreeView.ReadonlyIterator.nextValuePtr() -> *const Element.Type — list iteration that hands out per-element pointers as the depth-iterator walks the tree.
- BeaconState.validatorsPtrSlice(allocator) -> []*const Validator.Type — random-access pointer slice for callers that need sort / parallel workers / multi-pass.
The two APIs are complementary: iterator wins for single forward read passes (epoch_transition_cache.init, getEffectiveBalanceIncrementsZeroInactive); pointer slice wins for sort + random index access + parallel workers (epoch_cache.init calling syncPubkeys, slashings_cache.buildFromStateIfNeeded, upgrade_state_to_altair). 8 of 9 hot callers migrated; the last (upgrade_state_to_electra) keeps the value slice because its mutate-then-reread pattern would invalidate pointers.

Bench

bench_process_epoch and bench_process_block on mainnet era, fulu fork, slot 13336576 (2.18M validators), ReleaseFast, Apple Silicon. Both branches run with the same bench harness using c_allocator (no DebugAllocator overhead) for apples-to-apples comparison. *_total rows exclude the final hashTreeRoot (state-root recompute), which the bench tracks as its own segment.

Process epoch (segmented breakdown, ms/run averaged over 50 runs)

step	main	this branch	speedup
epoch_total	418.0	74.9	5.58×
`before_process_epoch`	193.8	39.9	4.86×
`inactivity_updates`	58.6	4.0	14.7×
`rewards_and_penalties`	80.8	14.2	5.69×
`effective_balance_updates`	67.2	5.7	11.8×
`proposer_lookahead`	16.5	10.9	1.51×

before_process_epoch (EpochTransitionCache.init) drops 4.86×: container_struct gives O(1) per-field reads on validators, and the nextValuePtr iterator skips the 263 MB clone that validatorsSlice used to do every epoch. inactivity_updates, rewards_and_penalties, effective_balance_updates get 5-15× from chunked_leaf making bulk reads/writes on Balances / InactivityScores / EpochParticipation SIMD-friendly + amortized CoW.

Process block (segmented breakdown, ms/run averaged over 50 runs)

step	main	this branch	speedup
block_total	166.1	67.3	2.47×
`operations`	162.9	64.0	2.55×
`block_header`	0.243	0.244	~same
`withdrawals`	0.021	0.021	~same
`execution_payload`	0.201	0.196	~same
`randao`	1.088	1.140	~same
`sync_aggregate`	1.675	1.492	~same

operations (bulk of block processing) gets 2.55× — chunked_leaf on the balance writes plus zero-copy validator reads in slashings_cache.buildFromStateIfNeeded. sync_aggregate's scattered sync-committee balance writes CoW a 2 KB ChunkedLeaf blob; its residual cost is BLS aggregate verification (~1.1 ms fixed, identical across branches).

Linux verification (AMD EPYC 9V74, 16 vCPU codespace, ReleaseFast)

Same fixture (mainnet era, fulu fork, slot 13336576, 2.18M validators), 50 runs/step. Speedup ratios reproduce on Linux/x86; absolute numbers are higher than Apple Silicon due to per-core differences.

Process epoch (segmented breakdown, ms/run)

step	main	this branch	speedup
epoch_total	666.0	123.0	5.41×
`before_process_epoch`	344.8	69.7	4.95×
`inactivity_updates`	67.6	4.4	15.4×
`rewards_and_penalties`	112.9	24.1	4.69×
`effective_balance_updates`	93.0	6.2	15.0×
`proposer_lookahead`	43.8	15.5	2.83×

Process block (segmented breakdown, ms/run)

step	main	this branch	speedup
block_total	350.8	110.7	3.17×
`operations`	346.0	106.4	3.25×

Process block (end-to-end fused, ms/run)

variant	main	this branch	speedup
`process_block` (with BLS)	49.3	38.5	1.28×
`process_block_no_sig`	10.63	2.79	3.81×

process_block_no_sig (BLS bypassed) drops 3.81× — the optimizations land cleanly on the non-BLS portion. The fused 1.28× reflects ~36 ms going to BLS aggregate signature verification per block, which is unaffected by PMT changes.

potentially fix #243

- Node is now union(enum) { free, zero, leaf, branch } with branch.root: ?[32]u8 encoding lazy/computed state instead of a bit-packed State enum. - Pool storage is MultiArrayList(NodeWithMeta) splitting Node and ref_count into separate columns (SoA-friendly for unref scans). - ref_count moves from 28-bit packed in State to standalone u32, max_ref_count raised from 0x1FFFFFFF to maxInt(u32) - 1. - Free list is encoded via the .free variant carrying next_free: Id, replacing the high-bit-of-state hack. - Public API surface unchanged: Pool methods and Id methods keep their signatures; StateView shim preserves predicate-based State semantics for existing callers. - proof.zig, View.zig, gindex.zig, ssz/* unchanged.

… tests Code review feedback on commit 940373f: - computeRoot: drop the 32 KB buf_a stack copy; read first-reduction pairs directly from slab.chunks. Saves 32 KB stack frame + 32 KB memcpy per call. - Storage: add doc comments on chunks (zero-tail invariant), len (semantic), and dirty (cache-coherence role). - Move slab tests to slab_test.zig matching the project's per-module test file convention; node_test.zig stays focused on Node tests.

len and dirty belong inline in the upcoming Node.slab union variant, not in heap Storage — keeping them in both places would force every CoW to sync two copies. Storage is now chunks-only (32 KB heap alloc per slab); per-slab metadata lives Node-side from B2 onward. Test for the removed dirty bitset assertion is dropped; remaining tests adjust their no-op .len assignment.

- Node now has a fifth variant: slab { chunks: [*]align(64), len: u16, dirty: StaticBitSet(K), root: ?[32]u8 }. - chunks is a many-pointer to a heap-allocated Slab.Storage; len/dirty/ root live inline in the union variant per the canonical design (Y is the layout we'll bench against in B6). - getRoot recovers *Slab.Storage via @ptrCast(@aligncast(chunks)) and caches the merkleized root in node.slab.root. - noChild treats slab as terminal (no Id-children). - StateView gains isSlab predicate. - Pool.unref does NOT yet free slab heap storage — that lands with Pool.createSlab in B3 (TODO marker placed).

- Pool.createSlab(chunks, len) heap-allocates Slab.Storage, copies the caller's chunks in, and seeds a slab Node with len/empty-dirty/null-root. - Pool.getSlabChunks / getSlabLen expose read-only access for downstream SSZ paths (Phase C). - Pool.unref now frees node.slab.chunks via Slab.destroy when refcount drops to zero (was a TODO(B3) placeholder); slab lifecycle is now fully Pool-managed. - Replace B2's white-box slot-planting test with three API-driven tests: create/get round-trip, unref-leak check, getRoot integration. After this commit, callers should never construct Node.slab variants manually — go through Pool.createSlab.

…riant - getSlabChunks / getSlabLen migrate from Pool methods to Id methods, matching Id.getLeft / Id.getRight conventions. - Asserts replaced with Error.InvalidNode returns: ReleaseFast strips std.debug.assert, exposing UB if a non-slab Id were passed; explicit errors are safe in all build modes. - Tests updated to the new call form: slab_id.getSlabChunks(&pool). Pool.createSlab stays a Pool method (it allocates from the Pool's allocator); only the read-side getters move to Id.

- Both methods clone the heap Slab.Storage, apply the requested chunk writes, and plant a fresh slab Node pointing at the new Storage; the receiver slab is unchanged (CoW). - New slab has root: null (lazy) and dirty bits set on every changed index. The Slab.Storage zero-tail invariant is preserved because we only overwrite chunks at indices < K (asserted) and never touch chunks[len..K] during the copy. - Both methods return Error.InvalidNode on non-slab Id (sibling-API consistency with Id.getSlabChunks/Len). - Tests: basic CoW invariant, len preservation, batch update, empty batch edge, error path.

Verifies that slab Ids appended to FillWithContentsIterator at the appropriate depth produce a merkleized root identical to a tree built from individual chunk leaves. Catches regressions in: - noChild semantics (slab as terminal) - getRoot integration of slab + branch lazy compute - Slab.computeRoot vs std merkleize equivalence 40 PMT tests passing (was 39).

…threshold When isBasicType(Element) and max_chunk_count >= 2 * Slab.K, FixedListType builds the SSZ tree via Pool.createSlab leaves at depth chunk_depth - k_log2 instead of pool.createLeaf at depth chunk_depth. Each slab packs K=1024 chunks worth of items contiguously; trailing chunks within the partial last slab remain zero-bytes (Storage zero-tail invariant preserved). Affected fields when used in BeaconState (limit-driven): balances, randao_mixes, slashings, participation, inactivity_scores, etc. - Re-export Slab from persistent_merkle_tree module. - comptime threshold: max_chunk_count >= 2 * Slab.K (== 2048 chunks). Smaller lists keep the existing per-chunk leaf path (regression-tested). - The slab path manually inlines an iterator-style merging loop because FillWithContentsIterator's zero filler hardcodes @enumFromInt(level), which gives the depth-level zero hash. With slab leaves each at depth k_log2, the correct filler is at depth (level + k_log2). C1.5 will factor this out by adding a leaf_offset parameter to the iterator. - Tests: large packed u64 list (item_count = 2*K*4 = 8192) verifies root vs std hashTreeRoot AND slab nodes present at boundary depth. Small list (limit=1024) confirms threshold gate keeps leaf path.

…ove 2*K threshold" This reverts commit b4484bb.

When each appended Id represents a non-leaf subtree (e.g., a Slab Id is a depth-k_log2 subtree of K chunks), the zero filler emitted by finish() for missing right siblings at iterator level L must be at absolute depth L + leaf_offset, not L. - initWithOffset(pool, depth, leaf_offset) is the new explicit constructor. - init(pool, depth) preserves the legacy depth-0-leaves contract by delegating to initWithOffset(..., 0). - append() unchanged (doesn't depend on leaf_offset). - finish() uses (level + leaf_offset) wherever it previously used (level). Tests: full-tree slab build matches per-leaf reference; partial-fill slab build matches per-leaf reference with 1/4 missing slabs zero-padded. This validates that finish()'s zero-filler emits zeros at the correct absolute depth, not iterator-level depth.

- Adds toValuePackedFromBytes / fromValuePackedIntoChunk to UintType and Bool element-tree APIs. These let slab-backed containers decode/encode packed items directly from chunk bytes, no Node.Id round-trip. - BasicPackedChunks gains a 4th comptime parameter use_slab. When true, get / set / getAllInto navigate to slab-boundary depth (chunk_depth - Slab.k_log2) and read/write chunks via Id.getSlabChunks / setSlabChunk rather than per-chunk Ids. populateAllNodes is a no-op in slab mode. - Existing callers (list_basic.zig, array_basic.zig) pass use_slab=false to preserve current semantics; no behavior change. - Re-exports Slab as persistent_merkle_tree.Slab so chunks.zig can reference Slab.K / Slab.k_log2 from its slab-mode comptime branch. This lays groundwork for FixedListType/FixedVectorType opts.slab in step 3.

Adds TypeOpts { slab: bool = false } parameter to FixedListType and FixedVectorType. When opts.slab=true and isBasicType(Element), the type's tree paths build / walk slab leaves instead of per-chunk leaves. - tree.fromValue / deserializeFromBytes use FillWithContentsIterator. initWithOffset(slab_depth, k_log2). tree.toValue / serializeIntoBytes walk slab Ids at slab_depth and read chunks via Id.getSlabChunks. - TreeView (list_basic / array_basic) propagates ST.opts.slab to BasicPackedChunks(.., use_slab=ST.opts.slab). - list_basic features that traverse at chunk_depth (iteratorReadonly, sliceTo) compile-error when ST.opts.slab=true. Use get/set/getAll/ serialize/deserialize/hashTreeRoot for slab-enabled types. - All existing FixedListType/FixedVectorType callers updated to pass .{} (default slab=false). Mechanical signature update across consensus_types/{phase0,altair,bellatrix,capella,deneb,electra,fulu}, ssz/{type,tree_view}/*, test/{fuzz,spec}/*. - Round-trip tests for slab-enabled list and vector verify hash equality against the leaf-path reference. Verification: - PMT 42/42 ✓ - SSZ 198/198 ✓ (was 194 + 4 new slab round-trip tests) - state_transition 85/85 ✓ (no regression — opt-in is comptime, no existing type opts in to slab, leaf path unchanged) State_transition green confirms the opt-in plumbing has zero leak into the leaf path. Existing BeaconState fields (balances, randao_mixes, slashings, etc.) keep their leaf-per-chunk behavior. Per-field slab opt-in (and bench-driven validation) is future work.

Adds bench/ssz/list_slab.zig comparing the leaf-default and opts.slab=true instantiations of FixedListType(UintType(64), 1<<20) on representative balances-scale workloads. Five scenarios: fromValue — build tree from a populated value fromValue+getRoot — build + compute root (cold lazy hashes) toValue — decode all items back from the tree bulkSet+getRoot — full-rewrite shape (epoch-rewards-like) Run with `zig build run:bench_list_slab -Doptimize=ReleaseFast`. Apple silicon ReleaseFast results (1M u64 items): workload leaf slab speedup fromValue 1M 15.4 ms 1.8 ms 8.5× fromValue+getRoot 1M 39.5 ms 17.6 ms 2.2× toValue 1M 20.2 ms 1.1 ms 18× bulkSet+getRoot 1M 40.4 ms 17.5 ms 2.3× Reading: slab path eliminates 250K leaf-node allocations (8.5× on fromValue), provides contiguous chunk reads for bulk decode (18× on toValue), and benefits from the fixed-shape K-leaf merkleization specialization on root computation (~2× on getRoot-bound workloads). The bulkSet+getRoot scenario is shaped after balances epoch-rewards writeback — full overwrite + recompute root — and shows ~2.3× end-to-end gain. Sufficient signal to consider opting balances and similar 1M-scale packed primitive lists into opts.slab=true in a follow-up change.

Switches phase0.Balances from FixedListType(Gwei, VALIDATOR_REGISTRY_LIMIT, .{}) to .{ .slab = true }, taking the chunked-leaf path for the most-touched packed primitive list in BeaconState. Also drops the inline duplicate of Balances in fulu.BeaconState — it now aliases phase0.Balances correctly, which was already the case in altair/bellatrix/capella/deneb/electra. Bench: zig build run:bench_process_epoch -Doptimize=ReleaseFast (mainnet state at slot 13336576, fulu fork, 2.18M validators) Per-step deltas (segmented breakdown, time/run): step leaf slab speedup rewards_and_penalties 267.8 ms 37.6 ms 7.1× effective_balance_updates 120.5 ms 9.5 ms 12.7× inactivity_updates 109.0 ms 105.6 ms flat participation_flags 0.74 ms 0.73 ms flat proposer_lookahead 16.6 ms 16.4 ms flat End-to-end epoch (non-segmented): leaf: 5.929 s slab: 5.504 s ( -7.2% ) Total epoch time gain is dominated by `before_process_epoch` (cache rebuild ~5.3 s) which slab doesn't touch. The targeted balance-bound operations get 7-13× speedups; this commit unlocks that for any caller that iterates rewards / effective-balance updates against `state.balances`. Verification: - state_transition: 85/85 ✓ - PMT 42/42 ✓ + SSZ 198/198 ✓ (unchanged from prior commits) Other slab candidates (inactivity_scores, participation, randao_mixes, slashings) are deliberately left at .{} — opt them in only after their specific consumer paths are verified slab-safe and bench shows wins.

… opts.slab=true Switches altair.InactivityScores and altair.EpochParticipation (used by both previous_epoch_participation and current_epoch_participation) to opts.slab=true. Same idea as phase0.Balances in 304386c. Also drops fulu.BeaconState's inline duplicates of these list types in favor of the altair aliases — bellatrix/capella/deneb/electra already referenced altair correctly; fulu was the only outlier. Bench: zig build run:bench_process_epoch -Doptimize=ReleaseFast (mainnet state at slot 13336576, fulu fork, 2.18M validators) End-to-end deltas (time/run, non-segmented epoch): config epoch leaf baseline (no slab) 5.929 s + Balances slab (304386c) 5.504 s + InactivityScores + EpochParticipation slab 0.557 s ← this commit 10.6× total speedup over leaf baseline. Per-step segmented breakdown (this commit vs leaf baseline): step leaf all-slab speedup before_process_epoch 5.30 s 0.555 s 9.6× inactivity_updates 109.0 ms 5.82 ms 18.8× rewards_and_penalties 267.8 ms 18.1 ms 14.9× effective_balance_updates 120.5 ms 6.65 ms 18.1× participation_flags 0.74 ms 0.001 ms ~700× proposer_lookahead 16.6 ms 15.6 ms flat The 9.6× speedup of before_process_epoch is the second-order effect: cache-rebuild calls getAll() on participation lists, which used to walk the full leaf-per-chunk tree path. Slab-enabling those lists collapses that work to contiguous chunk reads — a win that single-field opt-in (balances only) couldn't unlock. Verification: - state_transition: 85/85 ✓ - PMT 42/42 + SSZ 198/198 unchanged Remaining slab candidates: randao_mixes (FixedVector(Bytes32)) and slashings (FixedVector(Gwei)). Both are vectors not lists; randao_mixes also has Bytes32 element type which may or may not satisfy isBasicType — defer to a follow-up that explicitly verifies before opt-in.

When a slab-opted-in list/vector is initially empty (e.g. genesis state before any participation flags are set) or sparsely filled, the tree's content shape is the early-return zero-leaf form (chunk_count==0 → createBranch(@enumFromInt(chunk_depth), @enumFromInt(0))). Navigating this tree at slab_depth lands on a zero sentinel — not a slab — and the slab-path consumers blow up with Error.InvalidNode in getSlabChunks. A zero subtree at slab boundary is semantically an all-zero slab. Fix the slab consumers to handle this: - BasicPackedChunks.get: return std.mem.zeroes(Element). - BasicPackedChunks.set: materialize a fresh zero-filled slab before applying the write; intermediate slab is unrefed after setSlabChunk. - BasicPackedChunks.getAllInto: @Memset the output slice for the slab's worth of items. - list.zig tree.toValue (slab branch): out.items already initialised to default_value; skip the slab payload read. - list.zig tree.serializeIntoBytes (slab branch): @Memset the output. - vector.zig tree.toValue / serializeIntoBytes (slab branches): same shape as list. Repro: minimal-preset altair finality_rule_4 — pre-state's previous_epoch_participation has length 0 at genesis. tree.fromValue takes the early-return path. processBlock then attempts to push attestation flags via epoch_participation.set; navigation lands on zero sentinel slot 10 (zero hash at depth k_log2=10) instead of a slab Node, causing getSlabChunks to fail. Verification (post-fix): - PMT 42/42 ✓ + SSZ 198/198 ✓ + state_transition 85/85 ✓ - spec_tests minimal (all 9 shards): operations 1731/1828 (97 skipped), epoch_processing 727/727, sanity 557/557, rewards 380/380, transition 267/267, random 561/561, fork 314/314, finality 48/48, merkle_proof 19/19 — total 4604/4701 pass, 0 failed (97 skipped due to missing fork test data).

When BasicPackedChunks.set is called on a slab Id that no parent Branch references yet (rc==0), the slab is exclusively owned by the TreeView's children_nodes cache. Mutating the heap chunks in place is safe and avoids the 32 KB CoW clone per write. The first write per slab still hits the CoW path (rc>=1, owned by the persistent tree), publishing a fresh transient slab to the cache. Every subsequent write to the same slab finds rc==0 and writes a single byte directly into the heap chunks, accumulating dirty bits across writes and invalidating the cached root. For attestation processing in altair+ blocks (~committee_size validators × many attestations per block, all hitting the same participation slab), this collapses N × 32 KB writes into 1 × 32 KB clone + (N-1) × byte writes. Bench impact (process_block ReleaseSafe, mainnet state slot 13336576): step leaf slab CoW slab+in-place operations_no_sig 9.1 ms 210 ms 1.6 ms (5.7× vs leaf) process_block_no_sig 15.5 ms 214 ms 4.8 ms (3.2× vs leaf) operations 39.5 ms 234 ms 27.2 ms (1.4× vs leaf) process_block 42 ms 240 ms 34 ms (1.2× vs leaf) Process_epoch (ReleaseFast) per-step deltas remain or improve slightly: step slab CoW slab+in-place inactivity_updates 6.3 ms 5.4 ms rewards_and_penalties 19 ms 19 ms effective_balance_updates 7.7 ms 6.7 ms End-to-end: process_block 14× regression (slab CoW vs leaf) flips to 3.2× speedup; process_epoch keeps its order-of-magnitude advantage on the bulk-getAll cache-rebuild path. Verification: - PMT 42/42 ✓ + SSZ 198/198 ✓ + state_transition 85/85 ✓ Correctness sketch: rc==0 on a slab Node Id means no parent Branch has ref'd it yet. After Pool.createSlab or Id.setSlabChunk, the returned Id has rc==0 until Pool.rebind during commit raises it to 1. While in the TreeView's transient state (children_nodes cache, gindex in `changed`), the slab is observable only via the current TreeView and its heap chunks can be mutated without violating shared-tree invariants.

Replaces the slab Node variant's many-pointer + len encoding with a single `*Slab.Storage` pointer (Lighthouse milhouse's Arc<PackedLeaf> shape adapted to our Pool-managed ref counting). Storage is now self- contained: chunks + len in one heap allocation, owned uniquely by the slab Node Id whose ref count tracks parent Branch references. Variant before: slab: { chunks: [*]align(64) [32]u8, len: u16, root: ?[32]u8 } Variant after: slab: { storage: *Slab.Storage, root: ?[32]u8 } Side effects: - Removes every `@ptrCast(@aligncast(node.slab.chunks))` reconstruction in getRoot / unref / setSlabChunk / setSlabChunks. Each callsite now uses `node.slab.storage` directly. - Slab.Storage gains a `len: u16` field (was inline in the Node variant). allocZero initialises len = 0; createSlab / setSlabChunk* propagate. - Drops the unused `dirty: StaticBitSet(K)` field that was inflating the slab variant by 128 B for no read-side benefit (root invalidation via `root = null` already drives lazy recompute, and computeRoot always merkleizes the full K-leaf subtree). - Slab variant size drops from ~171 B (with dirty) to ~48 B, on par with the branch variant (41 B). The Node union sizes itself to whichever variant is largest. Verification: - PMT 42/42 ✓ + SSZ 198/198 ✓ + state_transition 85/85 ✓ - process_block ReleaseSafe still produces correct slab+in-place numbers (operations_no_sig 1.6 ms vs leaf 9.1 ms — 5.7×). Note: bench_process_block still SEGVs in ReleaseFast / ReleaseSmall on this branch, independent of this refactor (ReleaseSafe's runtime tag checks mask the UB). Suspected Zig 0.16 codegen issue on tagged union dispatch in deep recursion paths through getRoot. Filed as a follow-up to investigate / report upstream; doesn't block correctness or the ReleaseSafe perf story.

…Depth `unfinalized_parents_buf` was declared as `undefined` and never initialized. On iteration 0, the post-rebind cleanup loop reads garbage Optional bytes and may call `pool.unref` on an arbitrary Id — recursively decrementing its children's ref counts. When this hits a slab Id still referenced by a parent branch, the slab is destroyed prematurely (use-after-free). Manifested as SEGV in `bench_process_block -Doptimize=ReleaseFast` deep inside `Slab.computeRoot`, with a bogus storage pointer pattern matching two adjacent u32 values from a freed slot's `next_free` field. ReleaseSafe hid the bug because its `.free => unreachable` runtime check panics explicitly; ReleaseFast UB-eliminates the unreachable arm and silently misroutes `.free` slots into the `.slab` arm. Fix: - Initialize `unfinalized_parents_buf` to all-null so iter 0 sees no stale entries. - Replace `.free => unreachable` in `Id.getRoot` with `@panic` so any future use-after-free surfaces as a clean panic rather than UB-driven SEGV under ReleaseFast. Verified: - test:persistent_merkle_tree 42/42 - test:state_transition 85/85, 0 leaks (was 236) - test:ssz 198/198 - test:spec_tests fork mainnet 276/38 skip (matches pre-fix baseline) - bench process_block ReleaseFast: operations_no_sig 1.7ms, process_block 33ms (matches ReleaseSafe baseline; previously SEGV)

The branch and slab variants stored root as `?[32]u8` so that lazy (uncomputed) state was represented by `null`. The Optional encoding adds a 1-byte tag and forces every read site to unwrap via `.?`. Replace with a plain `[32]u8` field plus a global sentinel constant `lazy_sentinel = [_]u8{0xFF} ** 32`. SHA-256 outputs are cryptographically unlikely to equal this value (~1 in 2^256). The unwrap-`.?` pattern is gone and the code is simpler. Note: this does NOT shrink `@sizeOf(Node)`. The slab variant's `*Slab.Storage` pointer forces 8-byte union alignment, which already pads the union to 48 bytes regardless of the Optional. So this is a correctness/clarity refactor, not a perf change. The follow-up SoA-split refactor (decomposing the union into top-level NodeWithMeta fields) is where the 6× per-node-visit cache footprint reduction will come from. Verified: - test:persistent_merkle_tree 42/42 - test:state_transition 85/85, 0 leaks - bench process_epoch ReleaseFast: epoch_total 328ms (vs 324ms baseline, within noise)

The tagged-union NodeWithMeta forced every navigation visit to load 48 B (cache density 1.3 nodes/cache-line). Profile (sample, 30s, ReleaseSafe) identified `Node.Id.getNodesAtDepth` as the dominant hotspot (+42% / +11.6 percentage points share) of the regression vs the pre-refactor SoA u32-column layout. Replace `union(enum) Node` with five top-level fields on `NodeWithMeta` (left, right, root, cache, kind) plus ref_count, so MultiArrayList yields six dense columns. Hot navigation (`getNodesAtDepth`, `setNodes`, `setNodesAtDepth`, `getNode`, `setNode`, `truncateAfterIndex`, `fillToLength`, `DepthIterator.next`) now reads only `kind` (1 B) and one child Id (4 B) per visit — beating the pre-fecab80f layout's footprint and recovering the 2.8x regression plus extra headroom. The `cache: ?*anyopaque` column holds `*Slab.Storage` for slab nodes; future variants (validators struct cache, sync committee cache, etc.) reuse the same column via type-erased pointer cast — no schema changes needed. Free-list link reuses the `left` column (free slots have no other use for it), keeping the column count at 6. Bench (process_epoch ReleaseFast, fulu, 2.18M validators): before_process_epoch: 282ms -> 169ms (-40%) epoch_total: 328ms -> 215ms (-35%) vs upstream main cd45847 (leaf): epoch_total: 394ms -> 215ms (-46%, 1.84x faster) before_process_epoch: 190ms -> 169ms (-11%, faster than leaf) inactivity_updates: 54.5ms -> 5.3ms (-90%) rewards_and_penalties: 74.6ms -> 17.9ms (-76%) effective_balance_updates: 58.9ms -> 6.5ms (-89%) process_block ReleaseFast: equivalent to upstream main within noise (~33ms standalone, ~214ms segmented). No path regression. All invariants preserved: - unfinalized_parents_buf null-init in setNodes/setNodesAtDepth (UAF fix from ca5ac32) - .free => @Panic in getRoot (defense vs ReleaseFast UB elimination) - lazy_sentinel for branch/slab uncomputed roots (from 17ded4a) Tests: 42/42 PMT, 198/198 SSZ, 85/85 state_transition (0 leaks), 276/38skip fork shard mainnet (matches baseline).

Introduces a new persistent_merkle_tree variant `branch_struct` whose payload is a heap-allocated `BranchStructRef` (vtable + cached deserialized struct pointer). Storage piggy-backs on the existing SoA `cache` column (no pointer-packing) and root caching uses the existing `lazy_sentinel` mechanism — same one-arm dispatch shape as `.slab`. A new SSZ type `StructContainerType(ST)` and view `StructContainerTreeView(ST)` expose this layer: - tree.fromValue / toValue / serialize round-trip via the branch_struct slot (one Pool node per container, regardless of field count) - field reads/writes are O(1) struct accesses; commit lazily recomputes the root only when needed - get_root is dispatched through the BranchStructRef vtable so the Pool stays type-erased Switches phase0.Validator from FixedContainerType → StructContainerType. `validatorsSlice` now reads directly from the cached struct array instead of walking 2M × 8-leaf merkle subtrees. Bench (process_epoch ReleaseFast, fulu, 2.18M validators): before_process_epoch: 169 ms -> 68 ms (-60%) epoch_total: 215 ms -> 110 ms (-49%) vs upstream main cd45847 (no slab, no struct cache): epoch_total: 394 ms -> 110 ms (-72%, 3.6x faster) before_process_epoch: 190 ms -> 68 ms (-64%, 2.8x faster) inactivity_updates: 54.5 ms -> 5.1 ms (10.7x) rewards_and_penalties: 74.6 ms -> 16.2 ms (4.6x) effective_balance_updates: 58.9 ms -> 6.0 ms (9.8x) Bench (process_block ReleaseFast): block_total (segmented): 214 ms -> 111 ms (-48%) process_block standalone: 33.5 ms -> 31.6 ms (~6% faster within noise) Tests: 42/42 PMT, 199/199 SSZ (+1 StructContainer test), 85/85 state_transition (0 leaks), 276/38skip fork shard mainnet (matches baseline). Design notes (vs PR #232 reference): - Stores BranchStructRef* in the SoA `cache` column rather than packing into the Id columns — cleaner, no bit-packing. - Reuses lazy_sentinel for root caching (one-arm dispatch like slab) vs PR 232's two-state branch_struct_lazy/computed enum split. - Omits `to_tree` materialization vtable slot (no current consumers need single-leaf proofs through Validator subfields; trivial to add later).

…d-root leak Two related leaks identified by Codex review of the chunked-leaf branch: 1. `Pool.deinit` only freed the MultiArrayList columns. Slots with `.slab` or `.branch_struct` kind own a heap-allocated cache pointer (`*Slab.Storage` or `*BranchStructRef`); when callers tear down the pool without first unref'ing every root, those payloads were leaked. Walk all slots and free the cache payload for any kind that owns one before destroying the column buffers. 2. `StructContainerTreeView.getFieldRoot` allocated a fresh PMT slot via `ChildST.tree.fromValue(pool, &field_value)` just to read the cached root, but never unref'ed it. Each call leaked one Pool slot for the pool's lifetime — composite child types would also leak their heap payloads. Add a per-view `field_root_cache: [chunk_count][32]u8` stable backing store, copy the hash into it, and unref the temporary slot immediately so the returned `*const [32]u8` stays valid without leaking. Verified: PMT 42/42, SSZ 199/199, state_transition 85/85 (0 leaks).

Opaque PMT nodes (.branch_struct from StructContainerType, .slab from chunked-leaf packing) terminate `getLeft`/`getRight` traversal because their children are not stored as separate Ids. This blocked single- and compact-multi proofs through any path that crosses validators[i].field or balances[i] / inactivity_scores[i] / participation[i]. Add `Pool.materializeBranchStruct` and `Pool.materializeSlab`, both of which lazily build a temporary plain PMT subtree from the cached payload that proof traversal can navigate. Extend the BranchStructRef vtable with `to_tree`, implemented for StructContainerType.WrappedT via `FixedCT.tree.fromValue`. proof.zig gains an `isOpaqueNode` / `materializeOpaque` pair plus a deferred-unref ArrayList for the compact-multi recursive walk; createSingleProof keeps a single optional materialized root and rejects nested opaques with `error.NestedOpaque`. Verified by two new regression tests (`single proof: validators[0]. withdrawal_credentials` and `single proof: balances[0]`) in src/fork_types/any_beacon_state.zig — both fail before this commit with `Error.InvalidNode` from `getRight` on the opaque node and pass after. Why spec tests didn't catch this: - EF `merkle_proof` shard only covers `BeaconBlockBody.blob_kzg_commitment_merkle_proof` (4 cases) - EF `light_client` shard has BeaconState proofs (sync_committee, finality_root) but lodestar-z has no `light_client` runner yet (separate follow-up). Even with a runner, those paths don't descend into list elements (`validators[i]`/`balances[i]`), so they wouldn't hit our opaque-node bug. - The included tests construct the exact `validators.0.X` and `balances.0` paths that branch_struct and slab respectively block. Tests: PMT 42/42, SSZ 199/199, fork_types 9/9 (incl. 2 new), state_transition 85/85 (0 leaks), spec_tests fork mainnet 276/38skip (matches baseline). Bench process_epoch ReleaseFast: 121ms / 77ms (within current variance, no regression — opaque materialization only fires inside proof generation, never on epoch hot paths).

@this

Replace `pub const NodeWithMeta = struct {...}` with file-level fields and `const Node = @this();`. `MultiArrayList(NodeWithMeta)` becomes `MultiArrayList(Node)`. Pure syntactic refactor — no behavior change, no perf change. Aligns with main branch's idiom and removes the awkward `NodeWithMeta` vs `Node` naming distinction.

Replace separate `kind: NodeKind` (u8) and `ref_count: u32` columns with a single packed `state: State` u32 column. Layout matches main: - bit 31 set → free slot; bits 0..30 = next-free Id - bit 31 clear → in-use; bits 28..30 = kind, bits 0..27 = ref_count Free-list link migrates from the `left` column into `state` itself (matching the legacy union encoding). Branch-free `kind()` decoder avoids a conditional on every navigation visit. Saves 1 byte/node and consolidates the ref/unref hot path: a single column read replaces touching both kind and ref_count. External callers (chunks.zig, list.zig, vector.zig, proof.zig) updated to read `state[idx].kind()` / `state[idx].refCount()` accordingly. Bench impact: epoch -5%, block -5%, sync_aggregate -30% vs baseline (plus 1 B/node memory savings).

Collapse three columns (`left: Id`, `right: Id`, `cache: ?*anyopaque`) into a single `payload: u64` overloaded by kind: - branch : low 32 = left Id, high 32 = right Id - slab : full 64-bit `*Slab.Storage` pointer - branch_struct : full 64-bit `*BranchStructRef` pointer - free : low 32 = next-free Id (high 32 unused) Final SoA layout: 3 columns total (payload + root + state), down from 6. Memory savings vs baseline: 9 B/node total (1 B from State pack, 8 B here from cache + one of left/right). New helpers `packChildren`/`unpackLeft`/`unpackRight`/`payloadAsPtr`/ `ptrAsPayload` keep the encode/decode local. `slabStorage` and `branchStructRef` now read from the payload column. External slab CoW path in `chunks.zig` uses the new public `Id.getSlabStorageMut` helper instead of poking the column directly. Bench impact: roughly perf-neutral vs the State-only commit on process_epoch / process_block; significant wins on slab-heavy paths (set_nodes_randomly -28%). Validator-iteration (branch_struct heavy) holds steady because the pointer fits in a single u64 read. Note: payload-merged design assumes 64-bit usize. lodestar-z (and all ETH consensus clients) target only 64-bit platforms.

`Id.getState(pool)` now returns the packed `State` value (matching main's API) instead of a wrapper struct. The State enum already exposes all predicates (`isFree`, `kind`, `refCount`, `nextFree`); the StateView indirection became redundant after C2 re-introduced packed State. Add `isZero/isLeaf/isBranch/isSlab/isBranchStruct` to State for parity with main. `isBranchLazy` / `isBranchComputed` stay on Id because they read both state and root columns. Net -24 lines: removes a struct that just forwarded calls to State.

setChunkedLeaf wrote chunk data on the grow path (push into a later chunk of an already-materialized ChunkedLeaf) without updating the ChunkedLeaf `len`, so it stayed at its Path-1 initial value and the documented "chunks at indices >= len are zero" invariant was violated. No live impact — computeRoot hashes all K chunks and ignores `len`, and getChunkedLeafLen has no production callers — but the field drifted. (Codex review, P2.) `len` is a function of the container length, which only the view knows. `BasicPackedChunks.set` now takes `container_len` (list view passes its live length, vector view its fixed length) and derives the authoritative per-ChunkedLeaf valid-chunk count; Path 1/2/3 all set it. The chunks layer no longer infers length from write positions. Tests — a stale `len` slips past root-equivalence checks, so both assert `getChunkedLeafLen` directly: - list_basic unit test: push-grow across chunk + ChunkedLeaf boundaries, assert getChunkedLeafLen and the trailing-zero invariant. - fuzz_ssz_chunked_leaf_set: assert every ChunkedLeaf's len after commit.

Remove the `// ........` workload-section divider comments — section dividers per the project comment style. The bench-runner structs and the file-header workload list are self-describing.

The property test only did randomized `set` on a fixed-size list, so it never grew a ChunkedLeaf — and its root-equivalence assertion is blind to ChunkedLeaf.len regardless. It now: - mixes random `set` and `push` over a growable reference, - asserts every ChunkedLeaf.len and the trailing-zero invariant after each commit, - round-trips a single proof on the grown/mutated chunked_leaf list; prior proof tests all used fromValue-built trees.

The old comment implied `depth + leaf_offset` itself must stay within `0..max_depth-1`, which contradicts the `<= max_depth` assertion. Spell out that `depthi` tops out at `depth-1`, so the largest sentinel index used is `depth-1 + leaf_offset` and `depth + leaf_offset <= max_depth` keeps every `@enumFromInt(zero_depth)` in range.

The small-object lane default was std.heap.c_allocator, pulling libc into every Pool. Switch the InitOptions default to std.heap.smp_allocator so the pool is libc-free by default (aligns with the standalone pure-Zig goal). Perf-critical paths that were validated on c_allocator now pin it explicitly: the NAPI bindings pool and the chunked_leaf bench. Every other call site already passes .allocator explicitly, so behavior is unchanged in the current tree; the new default only affects future call sites.

wemeetagain

Reviewed Node.zig and chunks.zig especially. Looks good.

Reconcile chunked_leaf / container_struct / Pool-allocator work with the #377 memory-safety hardening that landed on main. Conflicts were in 6 files (Node.zig, node_test.zig, chunks.zig, list_basic.zig, list_composite.zig, vector.zig); resolved by keeping both sides' intent (chunked_leaf paths + #377 errdefer/zero-fill/ownership hardening). #377's imported OOM fault-injection tests exposed pre-#377 latent double-frees that #346 branched before and never received. Root cause: error-cleanup paths "undid" a `pool.ref` with `pool.unref`, but unref frees at rc 0, so a caller-owned rc-0 orphan got freed early and unref'd again. Fixes: - Node.zig: add `unrefUnsafe` (decRefCount without freeing); use it in createBranch's failure errdefer. - tree_view_state.zig: `deinitAfterInitFailure` (drop root's ref without freeing so a failed view init leaves root to the caller); guard clearChildrenNodesCache against already-freed cached child roots. - chunks.zig: deinitAfterInitFailure passthroughs for both chunk structs. - container.zig: createBranch-class fix in the two view inits. - list_basic.zig / list_composite.zig: use deinitAfterInitFailure on the init error path. - Adapt #377's merged-in tests to this branch's APIs (Pool.InitOptions, 3-arg FixedListType). zig build + full `zig build test` green; test:ssz 227/227, test:persistent_merkle_tree 48/48.

spiral-ladder

still in the middle of reviewing but a quick question, will come back again

spiral-ladder · 2026-05-28T12:33:07Z

+const hashing = @import("hashing");
+const hash = hashing.hash;
+
+pub const k_log2: u8 = 6;


Q: how was 6 chosen?

Good question.

6 ⇒ K = 2⁶ = 64 chunks/blob (2 KiB). k_log2 trades the fold's two opposing costs:
larger K folds more of the subtree (fewer Node.Ids, more SIMD lanes per root → faster
bulk build/read), smaller K is cheaper to mutate (each set memcpies the whole K × 32 B
blob). I benchmarked both layers.

1. Microbench sweep — bench_list_chunked_leaf, 1M-item List<u64>

chunked_leaf time/run (lower = better):

workload K=32 K=64 K=128 K=256

fromValue (build) 2.01 ms 1.36 ms 1.10 ms 1.20 ms

toValue (bulk read) 1.42 ms 1.26 ms 1.57 ms 1.33 ms

sparseSet (CoW) 0.68 ms 0.93 ms 1.30 ms 2.40 ms

batchedSparseSet (512/blk) 1.47 ms 2.50 ms 4.65 ms 8.75 ms

proof 7.9 µs 10.6 µs 16.4 µs 43.5 µs

2. End-to-end — mainnet 2.18M-validator state, 50 runs

metric k=6 (K=64) k=5 (K=32)

epoch (segmented) 109.2 ms 114.9 ms

block (segments), with BLS 76.0 ms 77.1 ms

process_block+root_no_sig (BLS stripped) 10.6 ms 8.9 ms

spiral-ladder · 2026-05-28T12:47:43Z

+inline fn noChildKind(node_id: Id, kind: NodeKind) bool {
+    return switch (kind) {
+        .leaf => true,
+        .zero => @intFromEnum(node_id) == 0,
+        .branch => @intFromEnum(node_id) == 0,
+        .free => true,
+        .chunked_leaf => true,
+        .container_struct => true,
+    };
+}


nit: style

Suggested change

inline fn noChildKind(node_id: Id, kind: NodeKind) bool {

return switch (kind) {

.leaf => true,

.zero => @intFromEnum(node_id) == 0,

.branch => @intFromEnum(node_id) == 0,

.free => true,

.chunked_leaf => true,

.container_struct => true,

};

}

inline fn noChildKind(node_id: Id, kind: NodeKind) bool {

return switch (kind) {

.leaf,

.free,

.chunked_leaf,

.container_struct,

=> true,

.zero, .branch => @intFromEnum(node_id) == 0,

};

}

spiral-ladder · 2026-05-28T12:48:14Z

+        // `noChild` guards prevent reaching here for these variants.
+        .leaf, .free => unreachable,
+        .chunked_leaf => unreachable,
+        .container_struct => unreachable,


these can be put in one case as well

Resolve bench/state_transition conflict from #324 (move state clones out of benchmark run functions): - process_block.zig: keep this branch's struct-form Node.Pool.init. - Port the chunked_leaf commit/state_root timings (process_block segmented and process_epoch) onto BenchState.cloned_cached_state. - Migrate ProcessBlockRootBench to #324's before_each/after_each hooks so the per-iteration clone is no longer counted in its timing.

unref tolerates already-freed slots (drop the panic) so a recoverable OOM during setNodes/setNodesAtDepth rollback no longer aborts the process. chunks setChunkedLeaf Path 3 errdefer-reclaims the CoW node + 2KB blob if setChildNode OOMs. Tests: later-iteration OOM-rollback (setNodes/setNodesAtDepth), Path-3 leak, and State packing-edge cases; add ArmOnSizeAllocator test helper. Review (spiral-ladder): document ChunkedLeaf.k_log2; fold noChildKind / childrenOf unreachable arms.

Spell out the fold tradeoff (larger K → faster bulk build/read, bigger CoW memcpy) and the benchmark basis: a k_log2 sweep in bench_list_chunked_leaf plus process_epoch/process_block — 64 wins the bulk-read-bound epoch and ties the BLS-dominated block. Addresses review on the magic constant.

…e skew (#394) ## Problem `main` CI is red: the `build & test` job fails to **compile** `test:state_transition`: \`\`\` src/state_transition/sync_committees_witness.zig:148:33: error: expected 1 argument(s), found 2 pub fn init(opts: InitOptions) Error!Pool { \`\`\` ## Root cause — a semantic merge conflict (merge skew) - **#346** (chunked-leaf) changed `Node.Pool.init(allocator, pool_size)` → `Node.Pool.init(opts: InitOptions)` (2 positional args → 1 options struct). - **#367** (`getSyncCommitteesWitness`) landed `sync_committees_witness.zig` in parallel, still calling the **old 2-arg** form: `Node.Pool.init(allocator, 500_000)`. Both PR branches were green because neither tree contained the *combination*: #346's branch didn't have `sync_committees_witness.zig` (it predates #367 and was never updated to the latest `main`), and #367's base still had the old `Pool.init`. Merging #346 into a `main` that already had #367 produced code git merged cleanly (different files, no textual conflict) but which no longer compiles. ## Fix One line — update the stale call site to the new `InitOptions` form (matching every other PMT test in the repo, both fields pinned to the testing allocator for leak tracking). `git grep` confirms this is the only remaining old-style call site. ## Verification `zig build test:state_transition` → **96/96 tests passed** (was: compile error). ## Prevention Consider enabling **"Require branches to be up to date before merging"** or a **GitHub merge queue** so PR CI runs against the real post-merge tree and catches this class of logical conflict that git can't see.

A post-merge deep review of #346 surfaced several memory-safety issues on or adjacent to the chunked-leaf / zero-copy changes. This addresses five: - Composite set/push/setValue ownership: make them caller-retains-on-failure (matching std/Ghostty). `chunks.set` no longer deinits the passed view on its own reservation OOM; `setValue`/`pushValue` carry an errdefer over the view they build. Fixes a double-free in `load_state`'s applyModifiedValidators / appendNewValidators, where the caller's errdefer and set's self-free both ran on the `ensureUnusedCapacity` OOM path. - ChunkedLeaf root recompute: `getRoot`'s `.chunked_leaf` arm uses a stack scratch + `computeRoot` instead of `computeRootAllocating`, removing the only `@panic("OOM")` in src/ (aborted the Node.js host on OOM) and the per-recompute malloc/free on the hashTreeRoot path. - sumTargetUnslashedBalanceIncrements: assert `participations.len == validators.len`; #346's pointer slice turned a cross-list OOB into a garbage-pointer dereference. - ContainerTreeView.deserialize: add the `errdefer pool.unref(root)` its two siblings already carry, so an init OOM no longer strands the deserialized subtree until Pool.deinit. - Delete dead `fillToLength`/`fillToDepth` (pool-corrupting on first use, zero callers, superseded by `fillWithContents`). Each fix has a red-green verified regression test (OOM / no-orphan / no-double-free / zero-alloc sweeps). ssz 230/230, persistent_merkle_tree 53/53, state_transition 96/96.

A post-merge deep review of #346 surfaced several memory-safety issues on or adjacent to the chunked-leaf / zero-copy changes. This addresses five: - Composite set/push/setValue ownership: make them caller-retains-on-failure (matching std/Ghostty). `chunks.set` no longer deinits the passed view on its own reservation OOM; `setValue`/`pushValue` carry an errdefer over the view they build. Fixes a double-free in `load_state`'s applyModifiedValidators / appendNewValidators, where the caller's errdefer and set's self-free both ran on the `ensureUnusedCapacity` OOM path. - ChunkedLeaf root recompute: `getRoot`'s `.chunked_leaf` arm uses a reused Pool scratch field + `computeRoot` instead of `computeRootAllocating`, removing the only `@panic("OOM")` in src/ (aborted the Node.js host on OOM) and the per-recompute malloc/free on the hashTreeRoot path. A Pool field rather than a stack buffer because getRoot recurses to tree depth (~47 for a mainnet validators path), and chunked_leaf is a recursion leaf so one shared scratch is always safe. - sumTargetUnslashedBalanceIncrements: assert `participations.len == validators.len`; #346's pointer slice turned a cross-list OOB into a garbage-pointer dereference. - ContainerTreeView.deserialize: add the `errdefer pool.unref(root)` its two siblings already carry, so an init OOM no longer strands the deserialized subtree until Pool.deinit. - Delete dead `fillToLength`/`fillToDepth` (pool-corrupting on first use, zero callers, superseded by `fillWithContents`). Each fix has a red-green verified regression test (OOM / no-orphan / no-double-free / zero-alloc sweeps). ssz 230/230, persistent_merkle_tree 53/53, state_transition 96/96.

A post-merge deep review of #346 surfaced several memory-safety issues on or adjacent to the chunked-leaf / zero-copy changes. This addresses seven: - Composite set/push/setValue ownership: make them caller-retains-on-failure (matching std/Ghostty). `chunks.set` no longer deinits the passed view on its own reservation OOM; `setValue`/`pushValue` carry an errdefer over the view they build. Fixes a double-free in `load_state`'s applyModifiedValidators / appendNewValidators, where the caller's errdefer and set's self-free both ran on the `ensureUnusedCapacity` OOM path. - ChunkedLeaf root recompute: `getRoot`'s `.chunked_leaf` arm uses a reused Pool scratch field + `computeRoot` instead of `computeRootAllocating`, removing the only `@panic("OOM")` in src/ (aborted the Node.js host on OOM) and the per-recompute malloc/free on the hashTreeRoot path. A Pool field rather than a stack buffer because getRoot recurses to tree depth (~47 for a mainnet validators path), and chunked_leaf is a recursion leaf so one shared scratch is always safe. - sumTargetUnslashedBalanceIncrements: assert `participations.len == validators.len`; #346's pointer slice turned a cross-list OOB into a garbage-pointer dereference. - ContainerTreeView.deserialize: add the `errdefer pool.unref(root)` its two siblings already carry, so an init OOM no longer strands the deserialized subtree until Pool.deinit. - Delete dead `fillToLength`/`fillToDepth` (pool-corrupting on first use, zero callers, superseded by `fillWithContents`). - ChunkedLeaf.computeRoot: assert chunks past `len` are zero. A violated trailing-zero invariant would silently hash stale data into a wrong (consensus-divergent) root. - getChunkedLeafPtr: assert the node is exclusively owned (refCount == 0) before handing out a mutable blob pointer; in-place mutation of a shared node corrupts every tree referencing it. Each fix has a regression test (OOM / no-orphan / no-double-free / zero-alloc sweeps; the two asserts checked non-vacuous by fault injection), plus a u8 chunked_leaf round-trip (EpochParticipation's config; all other chunked_leaf tests use u64). ssz 231/231, persistent_merkle_tree 53/53, state_transition 96/96.

…hainSafe#346) ## Motivation State transition is dominated by PMT operations on `BeaconState`'s large basic-element lists (`Balances`, `EpochParticipation`, `InactivityScores`, ~1.4M items each) and per-field tree access for struct-shaped containers (`Validator`). On a mainnet fulu state with 2.18M validators, these account for the bulk of `processEpoch` and `processBlock` runtime. ## Description Five PMT-level changes that compose: 1. **`u64` payload column** — collapses every node kind's payload (branch left/right Ids, chunked_leaf pointer, container_struct vtable pointer, free-list link) into one machine word. State packs `[free_bit:1 | kind:3 | ref_count:28]` in `u32`. Cache validity moves out of `kind` into a `0xFF…` sentinel in the `root` column. Hot tree-walk visits touch only `state` (1 B) + 4-8 B from `payload`, fitting one cache line. 2. **`chunked_leaf`** — opt-in via `opts.chunked_leaf=true` on `FixedListType` / `FixedVectorType`. Bottom `k_log2 = 6` levels of the chunks subtree fold into one `*ChunkedLeaf` heap blob holding K=64 chunks. For 1M-item `List<u64>` pool metadata drops ~64× (256K Node.Id → 4096 ChunkedLeaves + 4096 heap blobs). Bulk read/write get SIMD-batched root recomputation and amortized CoW (one 2 KB memcpy per dirty leaf instead of 6 path clones per dirty chunk). 3. **`container_struct`** (originally ChainSafe#232) — a node kind whose payload is `*ContainerStructRef` (vtable + caller-allocated `T`). Backs `StructContainerType` for `Validator` etc. Field access = O(1) struct read instead of per-field tree walk; `hashTreeRoot` calls type's cached `get_root` directly. 4. **Pool dual-allocator** — `Pool` keeps two allocators routed by allocation kind: `page_allocator` for the MultiArrayList node columns (one large, infrequent allocation), and `allocator` (default `c_allocator`) for every per-node out-of-line heap blob — `ContainerStructRef`, `WrappedT`, and the 2 KB chunked_leaf blobs. Page-per-alloc on the small lane wastes ~70 GB of virtual address space at 2.18M validators on macOS arm64 and thrashes the TLB; the bucket allocator packs them densely. `Pool.init` switched to options-struct shape (`Pool.init(.{})` for production defaults). This unblocks `serializeValidators` / `getEffectiveBalanceIncrementsZeroInactive` / `getSingleProof` binding tests at mainnet scale (24 s → ~500 ms, 50× speedup, equal to main). 5. **Zero-copy validator access** — completes the container_struct value chain. PR ChainSafe#232 added `pool.getStructPtr(node, T)` but no list-iteration API was built on top, so callers still cloned the full 263 MB validators slice per epoch transition. This PR adds: - `StructContainerType.tree.getValuePtr(node, pool) -> *const T` — direct typed pointer into the pool's container_struct payload. - `ListCompositeTreeView.ReadonlyIterator.nextValuePtr() -> *const Element.Type` — list iteration that hands out per-element pointers as the depth-iterator walks the tree. - `BeaconState.validatorsPtrSlice(allocator) -> []*const Validator.Type` — random-access pointer slice for callers that need sort / parallel workers / multi-pass. The two APIs are complementary: iterator wins for single forward read passes (`epoch_transition_cache.init`, `getEffectiveBalanceIncrementsZeroInactive`); pointer slice wins for sort + random index access + parallel workers (`epoch_cache.init` calling `syncPubkeys`, `slashings_cache.buildFromStateIfNeeded`, `upgrade_state_to_altair`). 8 of 9 hot callers migrated; the last (`upgrade_state_to_electra`) keeps the value slice because its mutate-then-reread pattern would invalidate pointers. ## Bench `bench_process_epoch` and `bench_process_block` on mainnet era, fulu fork, slot 13336576 (**2.18M validators**), ReleaseFast, Apple Silicon. Both branches run with the same bench harness using `c_allocator` (no DebugAllocator overhead) for apples-to-apples comparison. `*_total` rows exclude the final `hashTreeRoot` (state-root recompute), which the bench tracks as its own segment. ### Process epoch (segmented breakdown, ms/run averaged over 50 runs) | step | main | this branch | speedup | |------|------|-------------|---------| | **epoch_total** | **418.0** | **74.9** | **5.58×** | | `before_process_epoch` | 193.8 | 39.9 | **4.86×** | | `inactivity_updates` | 58.6 | 4.0 | **14.7×** | | `rewards_and_penalties` | 80.8 | 14.2 | **5.69×** | | `effective_balance_updates` | 67.2 | 5.7 | **11.8×** | | `proposer_lookahead` | 16.5 | 10.9 | 1.51× | `before_process_epoch` (`EpochTransitionCache.init`) drops 4.86×: container_struct gives O(1) per-field reads on validators, and the `nextValuePtr` iterator skips the 263 MB clone that `validatorsSlice` used to do every epoch. `inactivity_updates`, `rewards_and_penalties`, `effective_balance_updates` get 5-15× from chunked_leaf making bulk reads/writes on `Balances` / `InactivityScores` / `EpochParticipation` SIMD-friendly + amortized CoW. ### Process block (segmented breakdown, ms/run averaged over 50 runs) | step | main | this branch | speedup | |------|------|-------------|---------| | **block_total** | **166.1** | **67.3** | **2.47×** | | `operations` | 162.9 | 64.0 | **2.55×** | | `block_header` | 0.243 | 0.244 | ~same | | `withdrawals` | 0.021 | 0.021 | ~same | | `execution_payload` | 0.201 | 0.196 | ~same | | `randao` | 1.088 | 1.140 | ~same | | `sync_aggregate` | 1.675 | 1.492 | ~same | `operations` (bulk of block processing) gets 2.55× — chunked_leaf on the balance writes plus zero-copy validator reads in `slashings_cache.buildFromStateIfNeeded`. `sync_aggregate`'s scattered sync-committee balance writes CoW a 2 KB `ChunkedLeaf` blob; its residual cost is BLS aggregate verification (~1.1 ms fixed, identical across branches). ### Linux verification (AMD EPYC 9V74, 16 vCPU codespace, ReleaseFast) Same fixture (mainnet era, fulu fork, slot 13336576, **2.18M validators**), 50 runs/step. Speedup ratios reproduce on Linux/x86; absolute numbers are higher than Apple Silicon due to per-core differences. #### Process epoch (segmented breakdown, ms/run) | step | main | this branch | speedup | |------|------|-------------|---------| | **epoch_total** | **666.0** | **123.0** | **5.41×** | | `before_process_epoch` | 344.8 | 69.7 | **4.95×** | | `inactivity_updates` | 67.6 | 4.4 | **15.4×** | | `rewards_and_penalties` | 112.9 | 24.1 | **4.69×** | | `effective_balance_updates` | 93.0 | 6.2 | **15.0×** | | `proposer_lookahead` | 43.8 | 15.5 | 2.83× | #### Process block (segmented breakdown, ms/run) | step | main | this branch | speedup | |------|------|-------------|---------| | **block_total** | **350.8** | **110.7** | **3.17×** | | `operations` | 346.0 | 106.4 | **3.25×** | #### Process block (end-to-end fused, ms/run) | variant | main | this branch | speedup | |---------|------|-------------|---------| | `process_block` (with BLS) | 49.3 | 38.5 | 1.28× | | `process_block_no_sig` | 10.63 | 2.79 | **3.81×** | `process_block_no_sig` (BLS bypassed) drops 3.81× — the optimizations land cleanly on the non-BLS portion. The fused 1.28× reflects ~36 ms going to BLS aggregate signature verification per block, which is unaffected by PMT changes. potentially fix ChainSafe#243

…hainSafe#367 merge skew (ChainSafe#394) ## Problem `main` CI is red: the `build & test` job fails to **compile** `test:state_transition`: \`\`\` src/state_transition/sync_committees_witness.zig:148:33: error: expected 1 argument(s), found 2 pub fn init(opts: InitOptions) Error!Pool { \`\`\` ## Root cause — a semantic merge conflict (merge skew) - **ChainSafe#346** (chunked-leaf) changed `Node.Pool.init(allocator, pool_size)` → `Node.Pool.init(opts: InitOptions)` (2 positional args → 1 options struct). - **ChainSafe#367** (`getSyncCommitteesWitness`) landed `sync_committees_witness.zig` in parallel, still calling the **old 2-arg** form: `Node.Pool.init(allocator, 500_000)`. Both PR branches were green because neither tree contained the *combination*: ChainSafe#346's branch didn't have `sync_committees_witness.zig` (it predates ChainSafe#367 and was never updated to the latest `main`), and ChainSafe#367's base still had the old `Pool.init`. Merging ChainSafe#346 into a `main` that already had ChainSafe#367 produced code git merged cleanly (different files, no textual conflict) but which no longer compiles. ## Fix One line — update the stale call site to the new `InitOptions` form (matching every other PMT test in the repo, both fields pinned to the testing allocator for leak tracking). `git grep` confirms this is the only remaining old-style call site. ## Verification `zig build test:state_transition` → **96/96 tests passed** (was: compile error). ## Prevention Consider enabling **"Require branches to be up to date before merging"** or a **GitHub merge queue** so PR CI runs against the real post-merge tree and catches this class of logical conflict that git can't see.

🤖 I have created a release *beep* *boop* --- ## [1.0.0](v0.1.2...v1.0.0) (2026-08-19) ### Features * add `state.getBuildersLength()` binding ([#472](#472)) ([be2b5ab](be2b5ab)) * **beacon-node:** add block state cache and checkpoint datastore ([#452](#452)) ([2145faa](2145faa)) * bindings to `getExpectedWithdrawals` and native tweaks ([#350](#350)) ([f47bc66](f47bc66)) * **bindings:** add pubkey cache syncPubkeys ([#537](#537)) ([542779f](542779f)) * **bindings:** aggregate cached public keys by validator index ([#397](#397)) ([2f90603](2f90603)) * **bindings:** align `BeaconStateView` with `IBeaconStateView` ([#347](#347)) ([b8ec273](b8ec273)) * **bindings:** configurable pubkey cache growth step ([#481](#481)) ([133ef24](133ef24)) * **bindings:** expose more APIs for STF ([#444](#444)) ([7fe2609](7fe2609)) * **bls:** add small MSM for npoints < 32 ([#393](#393)) ([b430638](b430638)) * **blst:** use external buffers for blst operations ([#358](#358)) ([78e4678](78e4678)) * **ci:** conditionally publish bindings with tag ([#355](#355)) ([ea77919](ea77919)) * **clock:** add clock module for slot/epoch timing ([#354](#354)) ([385b077](385b077)) * **fork_choice:** add Prometheus metrics module ([#309](#309)) ([cbc9d8d](cbc9d8d)) * **forkchoice:** implement the forkchoice module ([#246](#246)) ([7c62a9b](7c62a9b)) * getSyncCommitteesWitness ([#367](#367)) ([ef77649](ef77649)) * implement `loadState` API and binding ([#165](#165)) ([f903519](f903519)), closes [#159](#159) * **metrics:** metrics bindings ([#455](#455)) ([dd41999](dd41999)) * migrate blst,pubkeys to use zapi js dsl ([#331](#331)) ([fcd26ca](fcd26ca)) * **pubkeys:** add getPubkeyBytes binding ([#555](#555)) ([4ca51cf](4ca51cf)) * publish ARM64 musl bindings ([#482](#482)) ([ac764c9](ac764c9)) * **shuffle:** add swap-or-not shuffling module and binding ([#559](#559)) ([c2db37c](c2db37c)) * split nextValue fn ([#464](#464)) ([b47faeb](b47faeb)) * support getLatestWeakSubjectivityCheckpointEpoch ([#366](#366)) ([dcf3883](dcf3883)) * update fulu deposit processing ([#442](#442)) ([064335c](064335c)) ### Bug Fixes * avoid set ([#484](#484)) ([2e25d97](2e25d97)) * better generation of rand scalar ([#388](#388)) ([74dce77](74dce77)) * **bindings:** accept `dontTransferCache` in processSlots for backward compatibility ([#460](#460)) ([65df5af](65df5af)) * **bindings:** check signature infinity by default ([#509](#509)) ([2f5f281](2f5f281)) * **bindings:** clean up failed async BLS work ([#527](#527)) ([1111b00](1111b00)) * **bindings:** free metrics writer on scrape failure ([#529](#529)) ([4c8d94a](4c8d94a)) * **bindings:** harden random aggregate scalars ([#528](#528)) ([8e89a63](8e89a63)) * **bindings:** log level for missing fields ([#435](#435)) ([08faf41](08faf41)) * **bindings:** misordering of print for cpu count ([#381](#381)) ([752a972](752a972)) * **bindings:** populate epoch participation for test fixtures ([#436](#436)) ([8dbdd2e](8dbdd2e)) * **bindings:** refcount Pool to fix teardown panic ([#352](#352)) ([23b2f68](23b2f68)) * **bindings:** roll back partial N-API initialization ([#491](#491)) ([31c5ebb](31c5ebb)) * **bindings:** size BLS thread pool by cgroup-aware CPU count ([#386](#386)) ([3ae9522](3ae9522)) * **bindings:** validate class types before unwrap ([#514](#514)) ([2fd2ad5](2fd2ad5)) * **bindings:** validate secret key hex length ([#517](#517)) ([136e415](136e415)) * **bls:** align PublicKey.uncompress validation with Signature.uncompress ([#508](#508)) ([5a8dbe9](5a8dbe9)) * **bls:** bound randomized aggregation inputs ([#548](#548)) ([779d0bf](779d0bf)), closes [#542](#542) * **bls:** clean up partial thread pool initialization ([#490](#490)) ([d55e598](d55e598)) * **bls:** convert pippenger scratch bytes to element counts ([#513](#513)) ([a12ca92](a12ca92)) * **bls:** enforce 32-byte signing roots ([#545](#545)) ([72fd308](72fd308)) * **bls:** make batch cardinality structural ([#547](#547)) ([a06d8b2](a06d8b2)) * **bls:** preserve aggregate outputs on failure ([#521](#521)) ([e0b6dd1](e0b6dd1)) * **bls:** reject empty keygen salts ([#524](#524)) ([d2a9c86](d2a9c86)) * **bls:** reject unknown BLST error codes ([#525](#525)) ([9e4a6ad](9e4a6ad)) * **bls:** size pairing buffers for 32-bit targets ([#531](#531)) ([dc64a27](dc64a27)) * **blst:** default signature infinity check to true if not provided ([#387](#387)) ([021cdcb](021cdcb)) * **build:** remove `zig-out` from `files` ([#360](#360)) ([c52af09](c52af09)) * **ci:** fix caching spec test version ([#439](#439)) ([96885a1](96885a1)) * dangling state pointer in loadOtherState ([#450](#450)) ([81cbd5f](81cbd5f)) * **epoch_cache:** compute missing `next_proposers` ([#447](#447)) ([0088a29](0088a29)) * **epoch_cache:** populate decision roots in afterProcessEpoch ([#453](#453)) ([4b70a5e](4b70a5e)) * export asyncAggregateWithRandomness through napi binding ([#371](#371)) ([1d04c2b](1d04c2b)) * harden memory safety across PMT, SSZ tree views, and state transition ([#377](#377)) ([d6f5897](d6f5897)) * improve atomic ordering in ThreadPool and NAPI init ([#310](#310)) ([4b0a1cc](4b0a1cc)) * interface compatbility with NativeBeaconStateView ([#445](#445)) ([89e13d1](89e13d1)) * missing deinits in loadOtherState ([#459](#459)) ([094d278](094d278)) * missing state commits ([#454](#454)) ([a432b55](a432b55)) * no-op when syncPubkeys run on a pk cache with shrinking validator set ([#432](#432)) ([ed05a99](ed05a99)) * param order in BeaconBlockBody ([#348](#348)) ([d8b9c06](d8b9c06)) * pendingConsolidations bindings ([#449](#449)) ([b9c497e](b9c497e)) * **pmt,ssz:** harden chunked-leaf and zero-copy tree-view memory safety ([#400](#400)) ([de50c53](de50c53)) * populate cache balances during rewards/penalties processing ([#474](#474)) ([5bf23dc](5bf23dc)) * re-expose sizes ([#369](#369)) ([64b81f3](64b81f3)) * remove `slashValidator` gating on active status ([#448](#448)) ([d319a0d](d319a0d)) * **ssz:** drop redundant default-init pass in fixed-list decode ([#468](#468)) ([0c757be](0c757be)) * **ssz:** publish child cache entries after lookup ([#565](#565)) ([21e78c9](21e78c9)) * state transition binding exports ([#456](#456)) ([895982c](895982c)) * **state-transition:** group-check signature sets ([#515](#515)) ([42774e9](42774e9)), closes [#502](#502) * **state-transition:** isolate epoch step cache mutations ([#535](#535)) ([a83741a](a83741a)) * **state-transition:** repair Pool.init call broken by [#346](https://github.com/ChainSafe/lodestar-z/issues/346)×[#367](https://github.com/ChainSafe/lodestar-z/issues/367) merge skew ([#394](#394)) ([b42944f](b42944f)) * various fixes around config ([#433](#433)) ([c4f082c](c4f082c)) ### Performance Improvements * **bindings:** drop TS BLS comparison benches and report benchmarks on PRs ([#552](#552)) ([c909c6f](c909c6f)) * **bls:** add cache-aware signature verifier ([#562](#562)) ([063857e](063857e)) * **bls:** bypass worker queue for small batches ([#553](#553)) ([3f8a6df](3f8a6df)) * **epoch:** replace AutoHashMap with array lookup in reward/penalty caches ([#286](#286)) ([e4e181b](e4e181b)), closes [#243](#243) * **pmt:** chunked-leaf packing for basic lists and container_struct ([#346](#346)) ([ba156c4](ba156c4)) ### Code Refactoring * allocate `AsyncAggRandData` in one obj ([#384](#384)) ([459750f](459750f)) * **bindings/pubkeys:** simplify allocation strategy for aggregate ([#518](#518)) ([b82750f](b82750f)) * **bindings:** rename blst Lifecycle to State ([#516](#516)) ([0a9c179](0a9c179)) * **bindings:** use zapi js.io() instead of local io module ([#469](#469)) ([2b34cc0](2b34cc0)) * **bindings:** wake only required number of workers ([#383](#383)) ([1db57f1](1db57f1)) * **bls:** allocations around VMAS ([#395](#395)) ([dfda58c](dfda58c)) * **bls:** clean up bls ([#398](#398)) ([e0f3b9b](e0f3b9b)) * **bls:** remove need for tracking results for verifyMultipleAggregateSignatures ([#389](#389)) ([6fe5c3f](6fe5c3f)) * **bls:** remove single-threaded fallback ([#390](#390)) ([e057713](e057713)) * **clock:** single public Clock; internalize SlotClock ([#463](#463)) ([fbab1fa](fbab1fa)) * make XXXDecisionRoot fns return `js.String` ([#342](#342)) ([aef4420](aef4420)) * move shuffle into swap_or_not_shuffle module ([#558](#558)) ([e56efb2](e56efb2)) * **pubkeys:** centralize the process-wide cache ([#522](#522)) ([dc9669d](dc9669d)) ### Miscellaneous Chores * avoid slow tests in AGENTS.md ([#546](#546)) ([c60f2a9](c60f2a9)) * bump zapi to include musl build ([#485](#485)) ([0b488cc](0b488cc)) * **ci:** pin github actions with sha hashes ([#507](#507)) ([167b8f5](167b8f5)) * deprecate unused blst APIs ([#575](#575)) ([7b547fa](7b547fa)) * **deps:** bump zapi v2.1.0 -> v2.2.0 ([#376](#376)) ([0c240d8](0c240d8)) * **deps:** bump zbuild ([#403](#403)) ([e2545de](e2545de)) * **deps:** compile blst with ReleaseFast ([#391](#391)) ([753a896](753a896)) * **deps:** update zapi to 3.1.0 ([#483](#483)) ([f3e5827](f3e5827)) * **deps:** use zapi v2.1.0 ([#372](#372)) ([88f403a](88f403a)) * disable gemini auto code review ([#382](#382)) ([63e42a4](63e42a4)), closes [#380](#380) * **docs:** add comments section in AGENTS.md ([#566](#566)) ([0c09750](0c09750)) * move state clones out of benchmark run functions ([#324](#324)) ([e4035de](e4035de)) * prepare 1.0.0 release ([#576](#576)) ([20b657b](20b657b)) * release v0.1.2-rc.3 ([#370](#370)) ([e4fc551](e4fc551)) * **release:** 0.1.2-rc.2 ([#365](#365)) ([7046128](7046128)) * **release:** v0.1.2-rc.10 ([#477](#477)) ([9a4fad5](9a4fad5)) * **release:** v0.1.2-rc.4 ([#373](#373)) ([09468f1](09468f1)) * **release:** v0.1.2-rc.5 ([#374](#374)) ([f344efa](f344efa)) * **release:** v0.1.2-rc.6 ([#375](#375)) ([bdf5b67](bdf5b67)) * **release:** v0.1.2-rc.8 ([#401](#401)) ([06f91c2](06f91c2)) * **release:** v0.1.2-rc.9 ([#404](#404)) ([6024800](6024800)) * remove merge transition code ([#359](#359)) ([09b175d](09b175d)) * remove stale epoch cache TODOs ([#534](#534)) ([27a547a](27a547a)) * rename era shortHistoricalRoot to shortEraRoot ([#473](#473)) ([c75a4d3](c75a4d3)) * **scripts:** build bindings with preset ([#434](#434)) ([a1b5ef7](a1b5ef7)) * silence debug log when used in release builds ([#486](#486)) ([c5377d7](c5377d7)) * support dev workflow ([#364](#364)) ([fcb9a78](fcb9a78)) * update gloas types to align with the latest specs ([#431](#431)) ([1f065b5](1f065b5)) * update spec test version to v1.7.0-alpha.11 ([#451](#451)) ([5875660](5875660)) * update spec-test-version: v1.6.0-beta.2 -> v1.7.0-alpha.10 ([#441](#441)) ([f932b1c](f932b1c)) * update zapi to 4.0.0 ([#571](#571)) ([de8e3fd](de8e3fd)) ### Documentation * document security threat model ([#557](#557)) ([e678b87](e678b87)) * more comprehensive AGENTS.md ([#520](#520)) ([c74b386](c74b386)) * **pkix:** document load provenance requirement ([#556](#556)) ([37e0aa2](37e0aa2)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

GrapeBaBa added 30 commits May 1, 2026 22:39

feat(pmt): add Slab module with K=1024 storage and computeRoot

940373f

Revert "feat(ssz): FixedListType.tree.fromValue builds slab leaves ab…

cbd9f94

…ove 2*K threshold" This reverts commit b4484bb.

GrapeBaBa requested a review from a team as a code owner May 19, 2026 07:59

GrapeBaBa added 4 commits May 19, 2026 16:53

style(bench): drop section-divider comments in list_chunked_leaf

cd392a0

Remove the `// ........` workload-section divider comments — section dividers per the project comment style. The bench-runner structs and the file-header workload list are self-describing.

GrapeBaBa force-pushed the gr/pmt-chunked-leaf branch from f829344 to 79975a6 Compare May 21, 2026 06:09

GrapeBaBa force-pushed the gr/pmt-chunked-leaf branch from 79975a6 to 0f0a78c Compare May 21, 2026 06:10

wemeetagain previously approved these changes May 26, 2026

View reviewed changes

GrapeBaBa dismissed wemeetagain’s stale review via e68bcff May 28, 2026 12:46

spiral-ladder reviewed May 28, 2026

View reviewed changes

GrapeBaBa added 3 commits May 28, 2026 22:54

wemeetagain approved these changes Jun 9, 2026

View reviewed changes

wemeetagain merged commit ba156c4 into main Jun 9, 2026
30 checks passed

GrapeBaBa mentioned this pull request Jun 9, 2026

fix(state-transition): repair Pool.init call broken by #346×#367 merge skew #394

Merged

GrapeBaBa mentioned this pull request Jun 12, 2026

fix(pmt,ssz): harden chunked-leaf and zero-copy tree-view memory safety #400

Merged

twoeths mentioned this pull request Jun 17, 2026

feat: model phase0 Validator as struct #232

Closed

wemeetagain mentioned this pull request Jun 24, 2026

feat: add StructContainerTreeView, model Validator as StructContainerType #247

Closed

2 tasks

github-actions Bot mentioned this pull request Jul 30, 2026

chore: release 1.0.0 #536

Merged

wemeetagain deleted the gr/pmt-chunked-leaf branch August 17, 2026 19:46

Navigation Menu

Uh oh!

perf(pmt): chunked-leaf packing for basic lists and container_struct - #346

perf(pmt): chunked-leaf packing for basic lists and container_struct#346
wemeetagain merged 89 commits into
mainfrom
gr/pmt-chunked-leaf

GrapeBaBa commented May 5, 2026 •

edited

Loading

Uh oh!

wemeetagain left a comment

Uh oh!

spiral-ladder left a comment

Uh oh!

spiral-ladder May 28, 2026

Uh oh!

GrapeBaBa May 29, 2026

Uh oh!

spiral-ladder May 28, 2026

Uh oh!

spiral-ladder May 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

workload	K=32	K=64	K=128	K=256
`fromValue` (build)	2.01 ms	1.36 ms	1.10 ms	1.20 ms
`toValue` (bulk read)	1.42 ms	1.26 ms	1.57 ms	1.33 ms
`sparseSet` (CoW)	0.68 ms	0.93 ms	1.30 ms	2.40 ms
`batchedSparseSet` (512/blk)	1.47 ms	2.50 ms	4.65 ms	8.75 ms
`proof`	7.9 µs	10.6 µs	16.4 µs	43.5 µs

metric	k=6 (K=64)	k=5 (K=32)
`epoch (segmented)`	109.2 ms	114.9 ms
`block (segments)`, with BLS	76.0 ms	77.1 ms
`process_block+root_no_sig` (BLS stripped)	10.6 ms	8.9 ms

Uh oh!

Conversation

GrapeBaBa commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Description

Bench

Process epoch (segmented breakdown, ms/run averaged over 50 runs)

Process block (segmented breakdown, ms/run averaged over 50 runs)

Linux verification (AMD EPYC 9V74, 16 vCPU codespace, ReleaseFast)

Process epoch (segmented breakdown, ms/run)

Process block (segmented breakdown, ms/run)

Process block (end-to-end fused, ms/run)

Uh oh!

wemeetagain left a comment

Choose a reason for hiding this comment

Uh oh!

spiral-ladder left a comment

Choose a reason for hiding this comment

Uh oh!

spiral-ladder May 28, 2026

Choose a reason for hiding this comment

Uh oh!

GrapeBaBa May 29, 2026

Choose a reason for hiding this comment

1. Microbench sweep — bench_list_chunked_leaf, 1M-item List<u64>

2. End-to-end — mainnet 2.18M-validator state, 50 runs

Uh oh!

spiral-ladder May 28, 2026

Choose a reason for hiding this comment

Uh oh!

spiral-ladder May 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

GrapeBaBa commented May 5, 2026 •

edited

Loading

1. Microbench sweep — `bench_list_chunked_leaf`, 1M-item `List<u64>`