feat(shuffle): add swap-or-not shuffling module and binding

[nazarhussain]

Summary

Adds a Zig swap_or_not_shuffle module and JS bindings for the shuffle APIs Lodestar uses, so ChainSafe/lodestar can drop @chainsafe/swap-or-not-shuffle.

• Shares one shuffle implementation between state_transition and the JS binding.
• Matches the upstream Rust/reference behavior for implemented paths.
• Returns explicit errors for degenerate inputs that the reference package would crash or hang on.
• Leaves async binding support for a future zapi release, since Lodestar does not use it.

Testing

• Ported upstream shuffle reference tests and existing lodestar-z shuffle vectors.
• Added Zig coverage for validation order, error paths, proposer index, and sync committee selection.
• Checked beta metrics against unstable; no performance regression found.

Follow-ups

• feat(shuffle): add PTC sampling #563 adds PTC sampling on top of this.
• Async shuffle bindings can be added later if a consumer appears.

AI assistance was used for drafting and implementation support.

[nazarhussain]

This will soon go away as we will support namepsace pub const and enum in the Zapi.

[lodekeeper]

Metrics check from beta vs unstable, sampled 2026-08-17 around 08:30 UTC.

Compared the PR-related beta build against same-network unstable over a 12h window, with a 24h memory trend check:

• Beta branch build: v1.46.0/nh/zig-shuffle-integration/8c3cfb0
• Unstable baseline: v1.46.0/ec59619
• Matching node types checked: hoodi sas, semi, super; mainnet mainnet-fcr, mainnet-super
• Note: beta-solo was not useful for this branch comparison; its lodestar_version sample is still v1.46.0/e2b315e.

Verdict: I do not see a performance regression from this beta deployment.

Health is clean:

• Head distance: beta is 0-1 slots from clock, same as unstable.
• Finality distance: 2 epochs on beta and unstable.
• Block processor queue: 0 currently; max over 12h was 0 except one beta-semi sample at 1.
• Reorgs over 12h match unstable exactly: hoodi 1, mainnet 7.

Block/state/epoch processing is broadly comparable:

• beta-sas is slightly slower than unstable-sas, but only around +4-7% on the main paths:
  • received -> state transition: 0.180s vs 0.168s
  • state transition: 0.089s vs 0.085s
  • received -> fully verified: 0.467s vs 0.448s
  • epoch transition: 0.986s vs 0.943s
• beta-semi is effectively flat: most paths around +1%, epoch transition around +5%.
• beta-super is faster than unstable-super on most paths:
  • received -> state transition: 0.124s vs 0.131s
  • received -> fully verified: 0.384s vs 0.418s
  • epoch transition: 0.863s vs 0.921s
• Mainnet is also fine:
  • beta-mainnet-fcr is modestly slower on a few paths, around +4-7%, with CPU only +5%.
  • beta-mainnet-super is mixed/flat; no broad slowdown.

Resources do not support a regression hypothesis:

• CPU is similar or lower on most matched nodes:
  • hoodi sas: 3.11 vs 3.19 cores
  • hoodi semi: 0.71 vs 0.69 cores
  • hoodi super: 1.05 vs 1.21 cores
  • mainnet-fcr: 0.55 vs 0.52 cores
  • mainnet-super: 0.58 vs 0.67 cores
• RSS is lower on beta for matched nodes in the 12h average, with the usual uptime caveat:
  • beta beacons are ~48.7h old; unstable hoodi beacons are ~56.4h old.
• V8 heap and external memory are flat over 24h.
• GC pause time is low, below 1% of wall time on all checked beta nodes.
• Event-loop p99 is comparable or better except a beta-semi max spike, but no sustained queue or CPU pressure follows it.

Shuffle/cache-specific signals look clean:

• lodestar_shuffling_cache_size is 4 everywhere.
• lodestar_shuffling_cache_miss_count increase is 0 on beta and unstable over 12h.
• lodestar_cp_state_cache_state_reload_shuffling_cache_miss_count increase is also 0.
• Shuffling cache hit counts are comparable to unstable.

PTC-specific caveat:

• vc_ptc_duties_count, vc_ptc_duties_epoch_count, and vc_published_payload_attestations_total are all 0 on both beta and unstable in this sample.
• So there is no direct PTC-duty latency sample yet; this check mostly validates the downstream beacon-node processing/resource behavior around the native shuffle integration.

Watch item: beta-sas has slightly more validator misses than unstable-sas over the sample (about 45 vs 4, miss ratio ~0.41% vs ~0.32%), but target/source hit rates remain above 99.5%, head correctness is close to unstable, and this is not backed by CPU/memory/queue pressure. I would keep an eye on it during soak, but I would not call it a regression from the current metrics.

Recommendation: continue soak, but from current beta vs unstable metrics this looks OK.

[spiral-ladder]

mostly doc related comments, i kinda am of the opinion that we should cut down or limit the amount of comments or the quality of AI comments because they tend to write in a very roundabout way. in an isolated PR it seems fine but over the course of many PRs our codebase comments will become meaningless

[wemeetagain]

LGTM

Worth noting, we don't actually use the async variants of any shuffling code in production in lodestar (and I don't see a future where we ever will use it -- we'd rather just lower the entire STF!).

[wemeetagain]

Just noticed @spiral-ladder 's comments, changing my review to red on his behalf

[wemeetagain]

We decided we don't need async functions here based on our current usage of this, save the zapi enhancements for a future release

[GrapeBaBa]

Why must do there conversion for the first three error? Seems we don't need next two functions if we don't do this

[lodekeeper]

Good point. With the async/zapi-enhancement work out of scope, I do not think we need the extra reference-message conversion here.

I simplified the binding so it surfaces the native Zig/zapi error names consistently instead of mapping the first shuffle errors to @chainsafe/swap-or-not-shuffle prose messages, and removed the now-unneeded helper path. The tests now assert InvalidSeedLength / InvalidNumberOfRounds for these cases.

I cannot push directly to ChainSafe/lodestar-z (lodekeeper only has read permission), so I pushed the signed patch here: lodekeeper@a8148ba

Verification:

• PATH=/tmp/zig-0.16.0:$PATH zig fmt --check bindings/napi/shuffle.zig src/swap_or_not_shuffle/root.zig
• PATH=/tmp/zig-0.16.0:$PATH sh -c 'git ls-files "*.zig" | xargs zig fmt --check'
• PATH=/tmp/zig-0.16.0:$PATH pnpm prepare-mainnet
• pnpm exec vitest run bindings/test/shuffle.test.ts (18 tests)
• pnpm exec biome check bindings/test/shuffle.test.ts
• pnpm lint exited 0 with existing warnings in bindings/src/index.d.ts

Note: plain zig fmt --check . still reports unformatted files under downloaded zig-pkg/ dependency sources after the local build, so I used the tracked-file format check above for the actual repo files.

[spiral-ladder]

Suggested change

	//! JS binding for the `swap_or_not_shuffle` module. Function names, argument
	//! order and error messages match `@chainsafe/swap-or-not-shuffle`.
	//! JS binding for the `swap_or_not_shuffle` module.

[spiral-ladder]

these tests look kinda different from what we have in that repo, where are these from?

[nazarhussain]

These tests were not just coming from that repo, it is mix of those and existing shuffling tests.

[spiral-ladder]

Q: why arena?

[nazarhussain]

Documented the usage in the comments.

Arena seems fit for this use case.

The caches below are written until deinit and never freed piecemeal,
so an arena keeps their many small allocations to one teardown.

[wemeetagain]

Looks good!

Can you simplify / shorten the PR description before merging to only the most important information?

[spiral-ladder]

Can you simplify / shorten the PR description before merging to only the most important information?

+1, im a bit tired of reading ai pr descriptions to be completely honest, i skipped this description entirely

[lodekeeper]

@wemeetagain @spiral-ladder agreed. I cannot edit Nazar's PR body from lodekeeper (the PR edit endpoint returns 404 for this account), but here is a shorter replacement body that keeps only the merge-relevant bits:

## Summary

Adds a Zig `swap_or_not_shuffle` module and JS bindings for the shuffle APIs Lodestar uses, so ChainSafe/lodestar can drop `@chainsafe/swap-or-not-shuffle`.

- Shares one shuffle implementation between `state_transition` and the JS binding.
- Matches the upstream Rust/reference behavior for implemented paths.
- Returns explicit errors for degenerate inputs that the reference package would crash or hang on.
- Leaves async binding support for a future zapi release, since Lodestar does not use it.

## Testing

- Ported upstream shuffle reference tests and existing lodestar-z shuffle vectors.
- Added Zig coverage for validation order, error paths, proposer index, and sync committee selection.
- Checked beta metrics against unstable; no performance regression found.

## Follow-ups

- #563 adds PTC sampling on top of this.
- Async shuffle bindings can be added later if a consumer appears.

AI assistance was used for drafting and implementation support.

[wemeetagain]

thanks @lodekeeper I'm just going with that