Skip to content

feat: implement PubkeyIndexMap - #1

Merged
twoeths merged 11 commits into
mainfrom
te/pubkey_index_map
Apr 22, 2025
Merged

feat: implement PubkeyIndexMap#1
twoeths merged 11 commits into
mainfrom
te/pubkey_index_map

Conversation

@twoeths

@twoeths twoeths commented Apr 21, 2025

Copy link
Copy Markdown
Member

Description

  • implement PubkeyIndexMap in zig
  • write C-ABI functions and build shared library
  • write Bun binding code, by loading the shared library
  • unit tests + CI + benchmark

@github-actions

github-actions Bot commented Apr 21, 2025

Copy link
Copy Markdown
Contributor

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 6551d02 Previous: null Ratio
get values - 1000 406.00 ns/op
get values - naive - 1000 617.00 ns/op
set values - 1000 416.00 ns/op
set values - naive - 1000 677.00 ns/op
get values - 1000000 1.3880 us/op
get values - naive - 1000000 1.8760 us/op
set values - 1000000 1.0490 us/op
set values - naive - 1000000 2.6510 us/op

by benchmarkbot/action

@twoeths

twoeths commented Apr 22, 2025

Copy link
Copy Markdown
Member Author

I run this branch on my laptop, got

test/perf/index.test.ts
  get/set
    ✔ get values - 1000                                                    7874016 ops/s    127.0000 ns/op        -    4142957 runs   1.32 s
    ✔ get values - naive - 1000                                            6060606 ops/s    165.0000 ns/op        -    3678292 runs   1.31 s
    ✔ set values - 1000                                                    7633588 ops/s    131.0000 ns/op        -    7731718 runs   2.44 s
    ✔ set values - naive - 1000                                            3690037 ops/s    271.0000 ns/op        -    1349665 runs  0.606 s
    ✔ get values - 1000000                                                 1564945 ops/s    639.0000 ns/op        -    1358041 runs   1.11 s
    ✔ get values - naive - 1000000                                         1048218 ops/s    954.0000 ns/op        -     902932 runs   1.01 s
    ✔ set values - 1000000                                                 1584786 ops/s    631.0000 ns/op        -     973566 runs  0.808 s
    ✔ set values - naive - 1000000                                        700770.8 ops/s    1.427000 us/op        -     247581 runs  0.404 s

which is faster than napi-rs (blst-ts)

get/set
    ✓ get values - 1000                                                    4115226 ops/s    243.0000 ns/op   x1.004    1633842 runs  0.606 s
    ✓ get values - naive - 1000                                            5208333 ops/s    192.0000 ns/op   x0.950    1935707 runs  0.606 s
    ✓ set values - 1000                                                    4016064 ops/s    249.0000 ns/op   x1.016    1070166 runs  0.404 s
    ✓ set values - naive - 1000                                            2898551 ops/s    345.0000 ns/op   x0.953    1304631 runs  0.606 s
    ✓ get values - 1000000                                                 1298701 ops/s    770.0000 ns/op   x0.905    1012602 runs  0.909 s
    ✓ get values - naive - 1000000                                         1658375 ops/s    603.0000 ns/op   x0.907     701857 runs  0.505 s
    ✓ set values - 1000000                                                 1287001 ops/s    777.0000 ns/op   x0.898     668077 runs  0.606 s
    ✓ set values - naive - 1000000                                        781250.0 ops/s    1.280000 us/op   x0.948     864027 runs   1.21 s

(note that the naive is not the same due to different benchmark framework used)

@twoeths
twoeths marked this pull request as ready for review April 22, 2025 10:17
@twoeths
twoeths merged commit cc2c135 into main Apr 22, 2025
Comment thread src/pubkey_index_map.zig

pub fn set(self: *PubkeyIndexMap, key: []const u8, value: Val) !void {
var fixed_key: Key = undefined;
@memcpy(&fixed_key, key);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note that I tried to build a HashMap with []u8 as key to avoid redundant @memcpy but it's even slower
persisted the work in te/pubkey_index_map_no_copy branch for later reference

// the same to std.AutoHashMap except for the hash() and eql() functions
const U8SliceHashMap = std.HashMap(Key, u32, U8SliceContext, default_max_load_percentage);
const hash_fn = std.hash_map.getAutoHashFn(Key, U8SliceContext);
const U8SliceContext = struct {
    pub fn hash(_: U8SliceContext, key: Key) u64 {
        return std.hash.Wyhash.hash(0, key);
    }

    pub fn eql(_: U8SliceContext, a: Key, b: Key) bool {
        return std.mem.eql(u8, a, b);
    }
};

wemeetagain pushed a commit that referenced this pull request Nov 12, 2025
wemeetagain pushed a commit that referenced this pull request Mar 12, 2026
lodekeeper-z referenced this pull request in lodekeeper-z/lodestar-z Mar 28, 2026
…mport pipeline

Replace the test-only onBlockFromProto shim with a new onBlockFromState
function that properly handles:
- Checkpoint extraction from CachedBeaconState
- Unrealized checkpoint computation via state_transition
- Proposer boost assignment for timely blocks
- Target root computation from state block_roots
- Store checkpoint tracking updates

This is the #1 critical fix: without proper checkpoint extraction,
fork choice never updates justified/finalized checkpoints from blocks.

Also fixes:
- isBlockTimely: use hardcoded 4s attestation due (was referencing
  non-existent BeaconConfig.getAttestationDueMs)
- getCheckpointPayloadStatus: stub Gloas-specific code until ForkSeq
  gets gloas variant
- getFieldRoot: fix getElement -> getFieldRoot API mismatch

🤖 Generated with AI assistance
lodekeeper-z referenced this pull request in lodekeeper-z/lodestar-z Mar 28, 2026
Defer BLS signature verification from inline gossip handler to the
processor's batch handler:

1. GossipHandler.onAttestation: when processor is available, allocate
   heap-owned QueuedAttestation with decoded data + SSZ bytes, enqueue
   to processor without doing inline BLS. Skip BLS only for the batched
   path; inline fallback still does BLS before import.

2. processorHandlerCallback: attestation_batch handler now does:
   - Batch BLS verification (currently per-item; TODO: true BatchVerifier
     multi-pairing when signature set extraction is implemented)
   - On batch failure, falls back to individual verification
   - Imports valid attestations to fork choice via onSingleVote

3. Single attestation handler added for the non-batched case.

This eliminates the #1 gossip performance bottleneck: 6400 individual
BLS pairings/slot → batched verification with ~10-50x speedup potential.

🤖 Generated with AI assistance
lodekeeper-z referenced this pull request in lodekeeper-z/lodestar-z Mar 28, 2026
Per the Engine API spec, blockNumber, gasLimit, gasUsed, timestamp,
baseFeePerGas, blob_gas_used, excess_blob_gas, and withdrawal fields
(index, validatorIndex, amount) must be QUANTITY (variable-length hex,
no leading zeros), not fixed-width DATA encoding.

Replace hexEncodeU64/hexEncodeU256 with hexEncodeQuantity/hexEncodeQuantityU256
in all encode functions: encodeExecutionPayloadV1/V2/V3/V4,
encodeWithdrawal, encodeDepositRequest, encodeWithdrawalRequest,
and encodePayloadAttributesV1/V2/V3.

Fixes critical issue #1 and medium issue ChainSafe#5 from REVIEW.md.

🤖 Generated with AI assistance
lodekeeper-z referenced this pull request in lodekeeper-z/lodestar-z Mar 28, 2026
- attestation_service: use signing_ctx.slots_per_epoch instead of
  hardcoded /32 in reorg handler (fix #1)
- sync_committee_service: add TODO comments noting SYNC_COMMITTEE_SIZE=512
  and SUBNET_COUNT=4 are mainnet-only; minimal preset differs (fix #2)
- validator: add TODO comment on hardcoded 256 EPOCHS_PER_SYNC_COMMITTEE_PERIOD (fix ChainSafe#3)
- attestation_service: rename getDutiesAtSlot(slot) -> allDuties() since
  the slot param was unused; update call site (fix ChainSafe#4)
- signing: add current_epoch param to voluntaryExitSigningRoot() instead
  of calling std.time.timestamp() internally; keeps function pure/testable (fix ChainSafe#5)
- validator: replace inline @import("doppelganger.zig") with dopple_mod
  alias already imported at top of file (fix ChainSafe#6)
- validator_store: change isRemote from *const to *ValidatorStore since
  it calls mutex.lock() which requires mutability (fix ChainSafe#7)

🤖 Generated with AI assistance
@wemeetagain
wemeetagain deleted the te/pubkey_index_map branch August 17, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant