Skip to content

updated duties using split endpoints - #16421

Merged
james-prysm merged 69 commits into
developfrom
new-update-duties
Jun 9, 2026
Merged

updated duties using split endpoints#16421
james-prysm merged 69 commits into
developfrom
new-update-duties

Conversation

@james-prysm

@james-prysm james-prysm commented Feb 26, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

Feature

What does this PR do? Why is it needed?

validator client calls split endpoints for grpc instead of the combined call(getdutiesv2) and has some optimizations swapping current and next duties instead of calling both each epoch. the split duties properly fixes the attestation vs proposer dependent roots for lookahead.

best way to test is to remove the fork if statement and call the split directly , rebuild the image then run in kurtosis

How to reproduce the reorg test for this PR

End-to-end test recipe for validating the split-duty-endpoint path on a local kurtosis devnet, including a forced dependent-root mismatch that exercises the checkDependentRoots update branches (equivalent to a reorg).

1. Kurtosis config

~/git/kurtosis/gloas-config-4node.yml:

participants:
  - el_type: ethrex
    el_image: ethpandaops/ethrex:glamsterdam-devnet-4
    el_extra_params:
      - --http.api=eth,net,web3,admin
    cl_type: prysm
    cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
    vc_image: gcr.io/offchainlabs/prysm/validator:latest
    supernode: true
    cl_extra_params:
      - --verbosity=debug
    vc_extra_params:
      - --verbosity=debug
    count: 4

network_params:
  fulu_fork_epoch: 0
  gloas_fork_epoch: 1
  seconds_per_slot: 6
  genesis_delay: 30

additional_services:
  - dora
  - tx_fuzz

global_log_level: debug

2. Build local images (arm64)

bazel build //cmd/beacon-chain:oci_image_tarball \
  --platforms=@io_bazel_rules_go//go/toolchain:linux_arm64_cgo \
  --config=release --nouse_action_cache && \
docker rmi gcr.io/offchainlabs/prysm/beacon-chain || true && \
docker load -i bazel-bin/cmd/beacon-chain/oci_image_tarball/tarball.tar && \
bazel build //cmd/validator:oci_image_tarball \
  --platforms=@io_bazel_rules_go//go/toolchain:linux_arm64_cgo \
  --config=release --nouse_action_cache && \
docker rmi gcr.io/offchainlabs/prysm/validator || true && \
docker load -i bazel-bin/cmd/validator/oci_image_tarball/tarball.tar

Switch linux_arm64_cgolinux_amd64_cgo for x86.

Precondition: kill any running enclave before rebuilding (docker rmi fails if the image is in use):

kurtosis enclave rm -f gloas-grpc

3. Start enclave

The latest ethereum-package main has a Starlark error (GpuConfig undefined in zkboost_launcher.star). Pin to the last good commit:

kurtosis run --enclave gloas-grpc \
  github.com/ethpandaops/ethereum-package@555e9b72f25bd42a9c13640faef4f2866e62bc21 \
  --args-file ~/git/kurtosis/gloas-config-4node.yml

4. Inducing a dependent-root mismatch

On a clean 4-node network, natural reorgs don't happen, so the mismatch branch in checkDependentRoots doesn't get exercised. Two options to force it:

Option A — temporary code hook (deterministic)

Add a one-shot force to validator/client/duties.go's checkDependentRoots (do NOT commit this):

import "strconv"

// At package level:
var (
    debugForcedPrev bool
    debugForcedCurr bool
)

// Inside checkDependentRoots, right after computing needsPrevUpdate/needsCurrUpdate:
if headSlotNum, parseErr := strconv.ParseUint(head.Slot, 10, 64); parseErr == nil {
    if primitives.Slot(headSlotNum) >= 96 && headSlotNum%32 == 20 {
        if !debugForcedPrev {
            debugForcedPrev = true
            needsPrevUpdate = true
            log.WithField("headSlot", head.Slot).Warn("DEBUG: forcing needsPrevUpdate=true")
        } else if !debugForcedCurr {
            debugForcedCurr = true
            needsCurrUpdate = true
            log.WithField("headSlot", head.Slot).Warn("DEBUG: forcing needsCurrUpdate=true")
        }
    }
}

This fires the prev-update branch once at slot 116 and the curr-update branch once at slot 148 — both post-gloas (fork epoch = 2, so slot 64+).

Rebuild validator, restart enclave, then monitor.

Option B — network partition (non-deterministic but closer to real)

Pause a beacon container for ~1 epoch across an epoch boundary:

docker pause cl-2-prysm-geth--<suffix>
sleep 180
docker unpause cl-2-prysm-geth--<suffix>

With only 4 validators this may not actually produce a reorg — the paused node usually just catches up to the same canonical chain. A genuine fork needs a proper network partition:

docker network disconnect <kurtosis-net> cl-1-prysm-geth--<suffix>
docker network disconnect <kurtosis-net> cl-2-prysm-geth--<suffix>
# wait 1-2 epochs
docker network connect <kurtosis-net> cl-1-prysm-geth--<suffix>
docker network connect <kurtosis-net> cl-2-prysm-geth--<suffix>

Option A is the recommended path for regression testing — deterministic and ~14 minutes to both branches firing.

5. Monitoring

After the enclave is running past slot ~148 (≈15 min post-start), grep vc-1-geth-prysm logs:

# Forced triggers and resulting UpdateDuties info logs (option A)
kurtosis service logs gloas-grpc vc-1-geth-prysm -a \
  | sed 's/\x1b\[[0-9;]*m//g' \
  | grep -E "DEBUG: forcing|Updated duties due to"

# The bug this PR fixes: count "no duties for validators" errors.
# Expect 0 after fix. Pre-fix produces ~130 errors per forced trigger.
kurtosis service logs gloas-grpc vc-1-geth-prysm -a \
  | sed 's/\x1b\[[0-9;]*m//g' \
  | grep -c "no duties for validators"

# Split endpoint call counts — should rise on every forced/real trigger.
kurtosis service logs gloas-grpc vc-1-geth-prysm -a \
  | sed 's/\x1b\[[0-9;]*m//g' \
  | grep -oE "BeaconNodeValidator/(GetAttesterDuties|GetProposerDutiesV2|GetSyncCommitteeDuties|GetPTCDuties|GetDutiesV2)" \
  | sort | uniq -c

6. Pass criteria

  • Two DEBUG: forcing WARN lines at slots 116 and 148.
  • Two Updated duties due to … dependent root change INFO lines, each ~1–1.6s after the corresponding force.
  • "no duties for validators" error count: 0.
  • Split endpoint counts increase by one full set per trigger (AttesterDuties / ProposerDutiesV2 / SyncCommitteeDuties / PTCDuties).
  • No unrelated validator errors (ignore the HEZE_/EIP7928_/VIEW_FREEZE_* yaml config-parse warnings — those are unknown future-fork fields in the kurtosis preset, not caused by this PR).

7. Cleanup

Revert the debug hook from Option A before committing. The only intended changes on this PR are the two v.clearDuties() removals in checkDependentRoots plus the new test.

Which issues(s) does this PR fix?

Fixes #

Other notes for review

Acknowledgements

  • I have read CONTRIBUTING.md.
  • I have included a uniquely named changelog fragment file.
  • I have added a description with sufficient context for reviewers to understand this PR.
  • I have tested that my changes work as expected and I added a testing plan to the PR description (if applicable).

@james-prysm james-prysm mentioned this pull request Mar 2, 2026
4 tasks
github-merge-queue Bot pushed a commit that referenced this pull request Mar 4, 2026
**What type of PR is this?**

Other

**What does this PR do? Why is it needed?**

just moving some functions around to reduce duties split pr. part of
#16421

**Which issues(s) does this PR fix?**

Fixes #

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).
github-merge-queue Bot pushed a commit that referenced this pull request Mar 9, 2026
**What type of PR is this?**

Other

**What does this PR do? Why is it needed?**

This PR refactors the way we store different validator duties into a
duties store for easier splitting of tasks and in a future pr processing
duties for split endpoints. This PR will reduce the number of changes
when we start calling the different endpoints introduced in
#16416

pr is part of #16421

**Which issues(s) does this PR fix?**

Fixes #

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
james-prysm and others added 3 commits March 9, 2026 21:22
Resolved all conflicts by preferring develop's duty store types
and patterns. The split endpoint feature from this branch will be
re-added once grpc-split-duties-apis merges into develop.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
syjn99 pushed a commit to syjn99/prysm that referenced this pull request Mar 13, 2026
**What type of PR is this?**

Other

**What does this PR do? Why is it needed?**

just moving some functions around to reduce duties split pr. part of
OffchainLabs#16421

**Which issues(s) does this PR fix?**

Fixes #

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).
syjn99 pushed a commit to syjn99/prysm that referenced this pull request Mar 13, 2026
…s#16479)

**What type of PR is this?**

Other

**What does this PR do? Why is it needed?**

This PR refactors the way we store different validator duties into a
duties store for easier splitting of tasks and in a future pr processing
duties for split endpoints. This PR will reduce the number of changes
when we start calling the different endpoints introduced in
OffchainLabs#16416

pr is part of OffchainLabs#16421

**Which issues(s) does this PR fix?**

Fixes #

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
Comment thread validator/client/duty_store.go Outdated
Comment thread validator/client/duty_store.go Outdated
Comment thread validator/client/duties.go
Comment thread validator/client/duty_store.go
@nalepae

nalepae commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Approved without re-running the kurtosis devnet on the latest state of this PR.

nalepae
nalepae previously approved these changes Jun 8, 2026
@james-prysm
james-prysm enabled auto-merge June 8, 2026 20:03
pull Bot pushed a commit to All-Blockchains/prysm that referenced this pull request Jun 8, 2026
**What type of PR is this?**

Feature

**What does this PR do? Why is it needed?**

- adding /eth/v1/validator/proposer_preferences POST endpoint, also
hooks up validator client
- adding `proposer_preferences` SSE event topic on /eth/v1/events


kurtosis setup
```
participants:
  - el_type: ethrex
    el_image: ethpandaops/ethrex:glamsterdam-devnet-4
    el_extra_params:
      - --http.api=eth,net,web3,admin
    cl_type: prysm
    cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
    vc_image: gcr.io/offchainlabs/prysm/validator:latest
    supernode: true
    count: 2
    cl_extra_params:
      - --subscribe-all-subnets
      - --verbosity=debug
    vc_extra_params:
      - --enable-beacon-rest-api
      - --verbosity=debug

  - el_type: ethrex
    el_image: ethpandaops/ethrex:glamsterdam-devnet-4
    el_extra_params:
      - --http.api=eth,net,web3,admin
    cl_type: prysm
    cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
    vc_image: gcr.io/offchainlabs/prysm/validator:latest
    validator_count: 63
    cl_extra_params:
      - --verbosity=debug
    vc_extra_params:
      - --enable-beacon-rest-api
      - --verbosity=debug

network_params:
  fulu_fork_epoch: 0
  gloas_fork_epoch: 2
  seconds_per_slot: 6
  genesis_delay: 40

additional_services:
  - dora

global_log_level: debug

```

depends on OffchainLabs#16421 + adding
some connection changes for ptc duties
for more perfect run otherwise you will see some orphans

**Which issues(s) does this PR fix?**

implements ethereum/beacon-APIs#608 changes
note: this pr does not implement deprecation or the get proposer
preferences endpoint

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
@james-prysm
james-prysm added this pull request to the merge queue Jun 9, 2026
Merged via the queue into develop with commit 4d8c5a9 Jun 9, 2026
23 checks passed
@james-prysm
james-prysm deleted the new-update-duties branch June 9, 2026 14:35
@james-prysm james-prysm mentioned this pull request Jun 9, 2026
4 tasks
pull Bot pushed a commit to All-Blockchains/prysm that referenced this pull request Jun 10, 2026
**What type of PR is this?**
Feature

**What does this PR do? Why is it needed?**

hooks up the ptc rest apis to the validator client while in rest mode
that is missing them

follow up to OffchainLabs#16421

testing

```
participants:
  # Stateless VC: requests block + envelope inline (include_payload=true) and
  # publishes SignedExecutionPayloadEnvelopeContents (blobs + KZG proofs).
  - el_type: ethrex
    el_image: ethpandaops/ethrex:glamsterdam-devnet-4
    el_extra_params:
      - --http.api=eth,net,web3,admin
    cl_type: prysm
    cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
    vc_image: gcr.io/offchainlabs/prysm/validator:latest
    supernode: true
    count: 1
    validator_count: 32
    cl_extra_params:
      - --subscribe-all-subnets
      - --verbosity=debug
    vc_extra_params:
      - --enable-beacon-rest-api
      - --stateless
      - --verbosity=debug

  # Stateful VC: BN caches the envelope (include_payload=false) and the VC
  # publishes the spec-wire SignedBlindedExecutionPayloadEnvelope.
  - el_type: ethrex
    el_image: ethpandaops/ethrex:glamsterdam-devnet-4
    el_extra_params:
      - --http.api=eth,net,web3,admin
    cl_type: prysm
    cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
    vc_image: gcr.io/offchainlabs/prysm/validator:latest
    supernode: true
    count: 1
    validator_count: 32
    cl_extra_params:
      - --subscribe-all-subnets
      - --verbosity=debug
    vc_extra_params:
      - --enable-beacon-rest-api
      - --verbosity=debug

network_params:
  fulu_fork_epoch: 0
  gloas_fork_epoch: 2
  seconds_per_slot: 6
  genesis_delay: 40

additional_services:
  - dora
  - spamoor

# spamoor generates load so payloads are non-empty: eoatx for normal EL txs,
# blobs to exercise the blob/KZG path through the stateless envelope Contents flow.
spamoor_params:
  spammers:
    - scenario: eoatx
      config:
        throughput: 10
    - scenario: blobs
      config:
        throughput: 2

global_log_level: debug

```

observe if ptcs are being sent/voted on

**Which issues(s) does this PR fix?**

Fixes #

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
pull Bot pushed a commit to All-Blockchains/prysm that referenced this pull request Jun 10, 2026
**What type of PR is this?**

 Bug fix


**What does this PR do? Why is it needed?**

- support for ssz on get and propose payload envelope and payload
envelope content
- query parameter for gossip validation on payload envelope

relies on OffchainLabs#16421 and
OffchainLabs#16306, needs some small
changes to hook ptc up, if ptc is no issue 16421 is mainly required

Testing
```
participants:
  - el_type: ethrex
    el_image: ethpandaops/ethrex:glamsterdam-devnet-4
    el_extra_params:
      - --http.api=eth,net,web3,admin
    cl_type: prysm
    cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
    vc_image: gcr.io/offchainlabs/prysm/validator:latest
    supernode: true
    count: 2
    cl_extra_params:
      - --subscribe-all-subnets
      - --verbosity=debug
    vc_extra_params:
      - --enable-beacon-rest-api
      - --stateless
      - --verbosity=debug

  - el_type: ethrex
    el_image: ethpandaops/ethrex:glamsterdam-devnet-4
    el_extra_params:
      - --http.api=eth,net,web3,admin
    cl_type: prysm
    cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
    vc_image: gcr.io/offchainlabs/prysm/validator:latest
    validator_count: 63
    cl_extra_params:
      - --verbosity=debug
    vc_extra_params:
      - --enable-beacon-rest-api
      - --verbosity=debug

network_params:
  fulu_fork_epoch: 0
  gloas_fork_epoch: 2
  seconds_per_slot: 6
  genesis_delay: 40

additional_services:
  - dora

global_log_level: debug

```

**Which issues(s) does this PR fix?**

partially fixes ethereum/beacon-APIs#580 depends
on shane-moore/beacon-APIs#10
implements ethereum/beacon-APIs#613

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: satushh <satushh@gmail.com>
Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
syjn99 added a commit to syjn99/prysm that referenced this pull request Jun 11, 2026
**What type of PR is this?**
Feature

**What does this PR do? Why is it needed?**

hooks up the ptc rest apis to the validator client while in rest mode
that is missing them

follow up to OffchainLabs#16421

testing

```
participants:
  # Stateless VC: requests block + envelope inline (include_payload=true) and
  # publishes SignedExecutionPayloadEnvelopeContents (blobs + KZG proofs).
  - el_type: ethrex
    el_image: ethpandaops/ethrex:glamsterdam-devnet-4
    el_extra_params:
      - --http.api=eth,net,web3,admin
    cl_type: prysm
    cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
    vc_image: gcr.io/offchainlabs/prysm/validator:latest
    supernode: true
    count: 1
    validator_count: 32
    cl_extra_params:
      - --subscribe-all-subnets
      - --verbosity=debug
    vc_extra_params:
      - --enable-beacon-rest-api
      - --stateless
      - --verbosity=debug

  # Stateful VC: BN caches the envelope (include_payload=false) and the VC
  # publishes the spec-wire SignedBlindedExecutionPayloadEnvelope.
  - el_type: ethrex
    el_image: ethpandaops/ethrex:glamsterdam-devnet-4
    el_extra_params:
      - --http.api=eth,net,web3,admin
    cl_type: prysm
    cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
    vc_image: gcr.io/offchainlabs/prysm/validator:latest
    supernode: true
    count: 1
    validator_count: 32
    cl_extra_params:
      - --subscribe-all-subnets
      - --verbosity=debug
    vc_extra_params:
      - --enable-beacon-rest-api
      - --verbosity=debug

network_params:
  fulu_fork_epoch: 0
  gloas_fork_epoch: 2
  seconds_per_slot: 6
  genesis_delay: 40

additional_services:
  - dora
  - spamoor

# spamoor generates load so payloads are non-empty: eoatx for normal EL txs,
# blobs to exercise the blob/KZG path through the stateless envelope Contents flow.
spamoor_params:
  spammers:
    - scenario: eoatx
      config:
        throughput: 10
    - scenario: blobs
      config:
        throughput: 2

global_log_level: debug

```

observe if ptcs are being sent/voted on

**Which issues(s) does this PR fix?**

Fixes #

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
syjn99 added a commit to syjn99/prysm that referenced this pull request Jun 11, 2026
**What type of PR is this?**

 Bug fix


**What does this PR do? Why is it needed?**

- support for ssz on get and propose payload envelope and payload
envelope content
- query parameter for gossip validation on payload envelope

relies on OffchainLabs#16421 and
OffchainLabs#16306, needs some small
changes to hook ptc up, if ptc is no issue 16421 is mainly required

Testing
```
participants:
  - el_type: ethrex
    el_image: ethpandaops/ethrex:glamsterdam-devnet-4
    el_extra_params:
      - --http.api=eth,net,web3,admin
    cl_type: prysm
    cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
    vc_image: gcr.io/offchainlabs/prysm/validator:latest
    supernode: true
    count: 2
    cl_extra_params:
      - --subscribe-all-subnets
      - --verbosity=debug
    vc_extra_params:
      - --enable-beacon-rest-api
      - --stateless
      - --verbosity=debug

  - el_type: ethrex
    el_image: ethpandaops/ethrex:glamsterdam-devnet-4
    el_extra_params:
      - --http.api=eth,net,web3,admin
    cl_type: prysm
    cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
    vc_image: gcr.io/offchainlabs/prysm/validator:latest
    validator_count: 63
    cl_extra_params:
      - --verbosity=debug
    vc_extra_params:
      - --enable-beacon-rest-api
      - --verbosity=debug

network_params:
  fulu_fork_epoch: 0
  gloas_fork_epoch: 2
  seconds_per_slot: 6
  genesis_delay: 40

additional_services:
  - dora

global_log_level: debug

```

**Which issues(s) does this PR fix?**

partially fixes ethereum/beacon-APIs#580 depends
on shane-moore/beacon-APIs#10
implements ethereum/beacon-APIs#613

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: satushh <satushh@gmail.com>
Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
pull Bot pushed a commit to wsnchristopher/prysm that referenced this pull request Jul 21, 2026
**What type of PR is this?**
 Feature


**What does this PR do? Why is it needed?**


OffchainLabs#16421 split validator-client duty fetching across gRPC/REST and
introduced **promotion** — instead of re-pulling both current and next
epoch every boundary, the cached next-epoch duties are promoted into the
current epoch and only the new next-epoch is fetched.

This PR finishes the two follow-ups OffchainLabs#16928 left open:

1. **Next-epoch duties are always optional.** Previously a next-epoch
*attester* fetch failure was a hard error that aborted the whole update
— disrupting the *current* epoch too. Now every next-epoch duty type
fails softly (flagged, not fatal), so a next-epoch problem never breaks
the current epoch.
2. **Failures are retried per-type, not bundled.** Previously any single
missing next-epoch duty forced a full re-pull of *all four* duty types
at the next boundary. Now the missing types are re-fetched
**individually, mid-epoch**, and merged in — so promotion stays cheap
and a single transient failure doesn't cascade.

A third, related change keeps the dependent-root (reorg) handling
consistent with the new soft-failure model.

## What changed

### Next-epoch duties optional
`missingNextDuties` gains a `missingNextAttester` bit. In both the
full-fetch and promotion paths, a failed next-epoch attester is logged
and flagged (not returned as an error), exactly like proposer/sync/ptc
already were. Current-epoch attester/proposer remain required.

### Per-type mid-epoch retry
- `RetryMissingNextDuties` re-fetches **only** the duty types flagged
missing and overlays them onto the existing next-epoch duties
(`overlayNextDuties`), leaving the types that already succeeded
untouched. If the **attester spine** itself is missing (the attester
assignment is what creates each validator's duty row, so without it
there are no rows to overlay onto), it rebuilds the whole next epoch
instead.
- `MaybeRetryMissingNextDuties` is what the run loop calls each
non-epoch-start slot. It runs the retry in its **own goroutine** so the
current slot's attestation/proposal isn't blocked, but only when there's
actually missing work (`needsNextRetry`) and no retry is already in
flight (`retryInFlight` CAS) — so we don't spawn goroutines for nothing
or pile them up.
- Writes are guarded by a store **revision** counter: if an
epoch-boundary or head-event update lands while a retry goroutine is
mid-fetch, the now-stale write is dropped (`replaceNextDuties` applies
only if the store is still at the revision the fetch was based on).

### Dependent-root / reorg handling
`checkDependentRoots` no longer treats an *unknown* (nil) current
dependent root as "needs update" — that state only arises after a soft
next-epoch attester failure, and forcing a full `UpdateDuties` on every
head event while it's nil is wasteful. Recovery is instead owned by the
epoch boundary and the per-slot retry. Current-epoch reorg detection
(the `prev` dependent-root path) is unchanged.

## Before / after

### Per-duty-type fetch failure

| Duty type | Current epoch fails | Next epoch fails — **before** | Next
epoch fails — **now** |
|---|---|---|---|
| **Attester** | hard error → keep cached duties, retry next tick |
**hard error → aborts the whole update, breaking the current epoch** |
soft: flag `missingNextAttester`; spine rebuilt by the mid-epoch retry |
| **Proposer** | hard error → keep cache | soft, but forced a full
4-type re-pull next boundary | soft: flag; overlaid by the mid-epoch
retry |
| **Sync** | soft (logged) | soft, forced a full 4-type re-pull | soft:
flag; overlaid by the mid-epoch retry |
| **PTC** | soft (logged) | soft, forced a full 4-type re-pull | soft:
flag; overlaid by the mid-epoch retry |

### Triggers → action

| Trigger | **Before** | **Now** |
|---|---|---|
| Epoch boundary, no failures + same validators + stable dep root |
promote next→current, fetch next epoch only | same |
| Epoch boundary, can't promote (first run / validator-set drift) | full
fetch (current + next, all types) | same |
| Epoch boundary, a next-epoch type failed last cycle | **forced full
re-pull of all 4 types** | **promote** — the gap was already filled
mid-epoch |
| Mid-epoch slot | *(nothing — wait for next boundary)* | **retry only
the missing next-epoch types**, in a goroutine off the slot critical
path; no-op when nothing is missing |
| Head event, dependent root changed | `UpdateDuties` | same |
| Head event, current dependent root **unknown (nil)** | **trigger
`UpdateDuties`** | **skip** — let the boundary + per-slot retry recover
|

**Which issue(s) does this PR fix?**

Fixes #OffchainLabs#16928

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
syjn99 added a commit that referenced this pull request Aug 10, 2026
Each source now replaces only its own set: the flag and URL keys are loaded at
startup, the poller replaces the URL set and the file watcher the file set, so a
poll can no longer revert keys owned by another source. The key file holds only
its own keys, because persisting the flag or URL keys would hand their ownership
to the file across a restart and a key dropped upstream would keep validating.

updateLock covers replacement and its notification for all three updaters, so
subscribers never observe two unions out of order, and the file watcher can no
longer read the key file while a keymanager API write is in progress.

The keymanager API writes only to the file set and reports spec-compliant
statuses:

- POST without a key file errors instead of claiming "imported", since there is
  no permanent storage to import into, and "imported" is only reported once the
  file write is flushed and closed.
- POST of a key already in the union, or repeated within the same request,
  reports "duplicate" whatever its owner.
- DELETE of a flag or URL owned key errors naming the owner, an unknown key
  reports "not_found", and neither ever produces a 404.

Polling and a key file can also coexist now: a poll swaps only the URL keys, so
it no longer has to be disabled when a key file is configured, and an initial
URL failure stays soft whenever polling is enabled.

Introduce source-based key set management for Web3Signer

Adds a keySets type that keeps one key set per source (flag, URL, file) with the
validating set as their union. Every key has exactly one owner, so a source may
only replace its own set, and a key present in several sets is owned by the
first source in precedence order. Readers get copies so no internal map or slice
leaks.

Move decodePublicKeys into a shared helper

The keymanager loads keys from the flag and the key file too, so decoding and
deduping hex public keys is no longer specific to the URL source. Adds a pubkey
alias for the raw BLS key type the rest of the change is built on.

Merge remote-tracking branch 'upstream/develop' into feat/web3signer-hot-key-reload

# Conflicts:
#	encoding/bytesutil/hex_test.go

REST VC: Make the SSZ publish path spec-conformant - fall back to JSON on 415 (#17311)

**What type of PR is this?**

> Bug fix

**What does this PR do? Why is it needed?**

Previously, `PostSSZ` method sent:

```
Accept: application/octet-stream;q=0.95,application/json;q=0.9
```

(presumably) copied from the `GetSSZ` pattern, where preferring SSZ is
correct.

For the `PostSSZ`, it's incorrect. `PostSSZ` is only used for publish
endpoints (e.g., VC submits a signed message to BN) and per beacon-APIs,
those **never** produce an SSZ response body - successful publish is
bodiless and errors are JSON. Client doesn't need to say that "I want
you to give me response encoded by SSZ" in these paths. Also it's worth
noting that every caller of `PostSSZ` just discards the body and header
like `_, _, err := PostSSZ`.

Here's the result for scanning every path item in `beacon-APIs`:

| endpoint | request | response |
|---|---|---|
| `POST /eth/v2/beacon/blocks` | json + octet-stream | json only |
| `POST /eth/v2/beacon/blinded_blocks` | json + octet-stream | json only
|
| `POST /eth/v1/beacon/pool/attestations` (v2) | json + octet-stream |
json only |
| `POST /eth/v1/beacon/pool/payload_attestations` | json + octet-stream
| json only |
| `POST /eth/v1/beacon/execution_payload_envelopes` | json +
octet-stream | json only |
| `POST /eth/v1/beacon/execution_payload/bid` | json + octet-stream |
json only |
| `POST /eth/v1/validator/proposer_preferences` | json + octet-stream |
json only |
| `POST /eth/v1/validator/aggregate_and_proofs` (v2) | json +
octet-stream | json only |
| `POST /eth/v1/validator/register_validator` | json + octet-stream |
*(bodiless)* |

So this PR does following:

- `PostSSZ` sets `Accept` with `application/json`. Remove
`Accept`/q-value parsing that was brought from `GetSSZ`.
- JSON fallbacks from REST VC side only key on `415`, not `406`. Make
multi-handler (introduced in #17075) surface `415` instead of `406`.
- `PostSSZ` only returns `error`.

**Which issue(s) does this PR fix?**

N/A. Related to
- #17256

**Other notes for review**

`406 Not Accpetable` and `415 Unsupported Media Type` are one of a
common misconception we have (I also had).

- `415`: Server-side is saying, "Hey I have no idea how to decode your
request body/data. I only accept `application/json` but you sent
`application/octet-stream`". This is the *only* case where the
client-side should send the data differently encoded (e.g., SSZ-encoded
to JSON).
- `406`: Server-side is saying, "Hey you told me to respond in
`application/octet-stream` but I cannot respond you with the data type."

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).
 delay next epoch duties call for post gloas  (#17268)

**What type of PR is this?**

Bug fix

**What does this PR do? Why is it needed?**

Post-Gloas (split-duties path only): the validator client no longer
blocks the
epoch-boundary slot on next-epoch duty fetches. At slot 0 it promotes
the cached
next→current duties in memory (or, when it can't promote, fetches only
the current
epoch); the next epoch's duties are fetched in the background from slot
SlotsPerEpoch/4 (slot 8 mainnet), at 6000 BPS into the slot — between
the Gloas
aggregate (5000) and PTC (7500) deadlines. The pre-Gloas combined-duties
path is
unchanged.

**Which issue(s) does this PR fix?**

Branch `delayed-next-duties` @ `4d7dcc1c86`. Baseline = `develop` @
`8ff82d7d54`
(the exact parent of the fix commit, so every A/B below isolates this
single commit).

## Devnet e2e A/B — slot-0 proposal time (develop VC vs fix VC)
One chain, two VC groups (in-run control; the change is validator-only,
so both
groups share the same beacon image). Nodes 1–2 = develop VC
(`8ff82d7d54`),
nodes 3–4 = fix VC (`4d7dcc1c86`), confirmed by each VC's version stamp.

Custom config (NOT a premade file — saved locally as `gloas-vc-ab.yml`):
```yaml
participants:
  - el_type: ethrex
    el_image: ethpandaops/ethrex:glamsterdam-devnet-5
    el_extra_params: [--http.api=eth,net,web3,admin]
    cl_type: prysm
    cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
    vc_image: gcr.io/offchainlabs/prysm/validator:develop-vc   # develop VC
    supernode: true
    cl_extra_params: [--verbosity=debug]
    vc_extra_params: [--verbosity=debug]
    count: 2
  - el_type: ethrex
    el_image: ethpandaops/ethrex:glamsterdam-devnet-5
    el_extra_params: [--http.api=eth,net,web3,admin]
    cl_type: prysm
    cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
    vc_image: gcr.io/offchainlabs/prysm/validator:fix-vc       # fix VC
    supernode: true
    cl_extra_params: [--verbosity=debug]
    vc_extra_params: [--verbosity=debug]
    count: 2
network_params:
  fulu_fork_epoch: 0
  gloas_fork_epoch: 1
  seconds_per_slot: 6
  genesis_delay: 30
additional_services: [dora]
global_log_level: debug
```
Measured over post-gloas boundaries (slots 32/64/96/128/160/192),
time-into-slot from
each VC's `Submitted new block` / `Submitted new attestations` log vs
slot start:

| signal | develop (vc1/2) | fix (vc3/4) | delta |
|---|---|---|---|
| **slot-0 block proposal** | **~0.52s** (0.49–0.55, n=4) | **~0.18s**
(0.16–0.19, n=2) | **fix ~0.34s faster** |
| duties gRPC handler latency | ~185 ms | ~193 ms | none (idle node) |

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
Parallelize per-element hash tree roots for large SSZ lists (#17317)

**What type of PR is this?**

Hash large lists in parallel and don't use a mutex for feature config
reads.

The feature reads happens every single loop in a hash and bottlenecks on
large lists.

Parallelizing alone was a
*regression* — 2.31s to 3.48s on the benchmark below. Also it's not the
GC: `GOGC=800` did not change things

```
  41.82s 46.62%  sync/atomic.(*Int32).Add
  20.61s 22.97%  sync.(*RWMutex).RUnlock   (cum)
   1.40s  1.56%  gohashtree._hash
```

Benchmark: a list of 2^20 one-byte byte-lists (`List[List[byte, 2^30],
2^20]`),
Apple M4 Pro, 14 cores, `-benchtime 5x`, identical benchmark run on
both sides:

| | develop | this PR | speedup |
|---|---|---|---|
| `SliceRoot` | 1887 ms | 468 ms | 4.0x |
| `SliceRootProgressive` | 318 ms | 90 ms | 3.5x |

Computing the leaf root dominates, this can be made better with hastree
trickery but probably not worth it.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add IndexedAttestationGloas and AttesterSlashingGloas (#17305)

# Add IndexedAttestationGloas and AttesterSlashingGloas

> Part of the gloas-devnet-7 stacked series; based on
`separate-proto-ssz-methods`, diff shown against it.

Gloas needs its own attestation-shaped types so they merkleize
independently of the Electra
counterparts, rather than having `BeaconBlockBodyGloas` share generated
SSZ/HTR code with Electra.
`AttestationGloas` already landed in stack_1 (#17258); this PR mirrors
that split for the slashing
path by adding `IndexedAttestationGloas` and `AttesterSlashingGloas`,
and repointing
`BeaconBlockBodyGloas.attester_slashings` at the new Gloas type.

Both new messages are wire-identical to their Electra forms, so this is
a type split rather than a
format change: SSZ encodings and hash tree roots are unchanged, the
conversion helpers are
re-typing plus a deep copy, and the Beacon API continues to serve (and
accept) the existing Electra
JSON shape.

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---

<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>

---------

Co-authored-by: Bastin <bastin.m@proton.me>
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
Add proto/prysm/wrappers for proto-typed HTR helpers (#17304)

# Add proto/prysm/wrappers for proto-typed HTR helpers

> Part of the gloas-devnet-7 stacked series; based on
`mark-dirty-bs-trie`, diff shown against it.

`encoding/ssz` is meant to be a low-level merkleization package, but a
handful of its hash-tree-root
helpers take concrete generated proto types (`*ethpb.Fork`,
`*ethpb.Checkpoint`,
`*enginev1.Withdrawal`, `*enginev1.DepositRequest`,
`*enginev1.WithdrawalRequest`, and raw
transaction bytes). That forces `encoding/ssz` to import
`proto/prysm/v1alpha1` and
`proto/engine/v1`, which in turn depend on `encoding/ssz` transitively
through
`consensus-types/primitives`. Beyond the latent import-loop risk, the
arrangement causes trouble at
codegen time: if protobufs for types with helpers in the package are
regenerated, but helpers haven't
yet been modified to update their references to the generated types,
after the protobuf codegen run passes,
the helpers will have syntax errors, preventing the package from
compiling and being importable into the
codegen toolchain.

This PR moves those helpers into a new `proto/prysm/wrappers` package,
which is free to depend on
both `encoding/ssz` and the generated proto packages. After the move,
the `encoding/ssz` library
target has no proto dependencies at all. The helpers themselves are
unchanged in behavior — they
still delegate to the same `ssz.SliceRoot`, `ssz.SliceRootProgressive`,
`ssz.ByteSliceRoot`,
`ssz.ByteSliceRootProgressive`, and `ssz.BitwiseMerkleize` primitives
with the same limits — so
hash tree roots are identical before and after.

### Tests

- The proto-typed tests moved out of `encoding/ssz/htrutils_test.go`
into
`proto/prysm/wrappers/htr_test.go` with their expected-root fixtures
intact: `TestTransactionsRoot`,
`TestTransactionsRootProgressive`, `TestForkRoot`, `TestCheckPointRoot`,
`TestWithdrawalRoot`,
`TestWithrawalSliceRoot`, `TestWithdrawalSliceRootProgressive`,
`TestDepositRequestsSliceRoot`,
  `TestWithdrawalRequestSliceRoot`.
- `encoding/ssz/htrutils_test.go` keeps only the primitive tests
(`TestUint64Root`,
`TestByteArrayRootWithLimit`, `TestSlashingsRoot`, `TestByteSliceRoot`,
  `TestPackByChunk_SingleList`).
- `TestWithdrawalSliceRoot_ProgressiveSSZGate`, which drove the
fixed/progressive choice through the
feature flag and `runtime/version`, is replaced by
`TestWithdrawalSliceRootProgressive` calling
`wrappers.WithdrawalSliceRootProgressive` directly. The flag-gated
selection is still covered at
  its call sites (for example `TestHashTreeRoot_ProgressiveSSZGate` in
  `beacon-chain/state/state-native/progressive_ssz_test.go`).
- `encoding/ssz/htrutils_fuzz_test.go`'s `FuzzForkRoot` now fuzzes
`wrappers.ForkRoot`.
- Test call sites in
`beacon-chain/core/blocks/{payload,withdrawals}_test.go`,

`beacon-chain/rpc/prysm/v1alpha1/validator/{proposer,proposer_bellatrix}_test.go`,
and
`beacon-chain/light-client/lightclient_test.go` updated to the new
package.

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---

<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>

Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
Mark state fields dirty on early return (#17303)

# Mark state fields dirty on early return

`ApplyToEveryValidator` and `DecreaseWithdrawalBalances` mutate the
multi-value validator and
balance slices in place as they iterate, but they only recorded their
dirty field indices *after*
the loop ran to completion. Any error part way through returns early and
skips that bookkeeping, so
mutations that were already applied to the state end up untracked. The
next `HashTreeRoot` then
reuses stale cached subtrees for those fields and computes a root that
does not match the state's
actual contents.

The fix moves the dirty-field bookkeeping into a `defer` so it runs on
every exit
path, and in `ApplyToEveryValidator` record an index only once its
update has actually succeeded. No new tests accompany the change; it is
a bookkeeping correction on error paths that the existing state setter
tests do not exercise.

(Kasey is opening this PR stack to streamline the review process but the
author is @Inspector-Butters).

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---

<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>

Co-authored-by: Bastin <bastin.m@proton.me>
Implement the active-active mode in the REST validator client. (#17075)

**What type of PR is this?**
Feature

Depends on:
- https://github.com/OffchainLabs/prysm/pull/17143

### The active-passive connection scheme
Currently, a Prysm validator client can connect to multiple beacon nodes
via REST, but only one node is actively used for SSE events and HTTP
requests at any given time. This is called the **active** beacon node,
while the others are **passive**.

If the active node fails or goes offline, the validator client
automatically switches to one of the passive nodes, which then becomes
active. This is the **active-passive** connection scheme.

While this approach generally works well, it has limitations in certain
situations. Consider a validator client connected to BN-1, BN-2, and
BN-3, with only BN-1 active. If BN-1 experiences issues (such as poor
peering or slow block execution due to disk constraints) and fails to
import a block within the 4-second deadline, while BN-2 and BN-3
successfully import it, the validator client faces a problem.

Because the validator client only listens to BN-1's SSE events, it will
not receive the head event for the new block before the deadline.
Consequently, when it requests attestation data from BN-1, it receives
stale data corresponding to the previous block. The validator client
signs and broadcasts this attestation through BN-1, which means the
validator misses the head vote, and potentially the source and target
votes if this is the first block of an epoch.

This situation could have been avoided if the validator client could
listen to SSE events from all connected beacon nodes (not just the
active one) and request fresh attestation data from any of them (not
just the active one).

Here enters the **active-active** connection scheme.

### The active-active connection scheme
With the **active-active** connection scheme, a validator client
actively listens to and requests data from all connected beacon nodes.
When responses differ between beacon nodes, the validator client selects
the best one based on the context.

This pull request implements the **active-active** connection scheme.

To implement this connection scheme, we introduce four new concepts:

**The multi-event stream**
The validator client needs to listen to SSE events from multiple beacon
nodes. The multi-event stream:
- connects to (and reconnects to, if needed) each beacon node's event
stream,
- merges these streams into a single stream,
- removes potential duplicates, and
- sends the result to an output stream.

```
host-1 --\
host-2 ---+--> merged --> deduper --> out
host-3 --/
```

**The multi-handler**
The multi-handler is a framework that defines how HTTP requests are sent
to connected beacon nodes and which response is selected. It provides
the following options:

- `WithRace`: Should requests be sent sequentially or concurrently to
the beacon nodes?
- `WithAccept/WithSSZAccept`: Should all 2xx responses from any beacon
node be accepted, or should additional acceptance criteria be applied?
- `WithDeadline`: How long should the validator client wait if no
acceptable response is returned?
- `WithRepoll`: Should the validator client retry if no acceptable
response is returned?

`WithRepoll` takes a **re-poll mode** that decides what stops the
re-polling:
- `UntilAccepted`: keep re-polling until an *accept-passing* (fresh)
response arrives, or the deadline fires.
- `UntilAny2xx`: re-poll only while *no usable response at all* has
arrived, stopping as soon as any node returns one.

**The freshness options**
Now that we've implemented the multi-handler, we need to define how to
use it (which options to set) for the 4 objects that require it.

| Function | Endpoint | Object requested |
|---|---|---|
| `attestationFreshnessOptions` |
[/eth/v1/validator/attestation_data](https://ethereum.github.io/beacon-APIs/#/Validator/produceAttestationData)
| `AttestationData` |
| `syncCommitteeFreshnessOptions` |
[/eth/v1/beacon/blocks/head/root](https://ethereum.github.io/beacon-APIs/#/Beacon/getBlockRoot)
| `BlockRootResponse` |
| `blockFreshnessOptions` |
[/eth/v3/validator/blocks/{slot}](https://ethereum.github.io/beacon-APIs/#/Validator/produceBlockV3)
(or
[/eth/v4/validator/blocks/{slot}](https://ethereum.github.io/beacon-APIs/?urls.primaryName=dev#/Validator/produceBlockV4)
for Gloas) | `BeaconBlock` |
| `payloadAttestationFreshnessOptions` |
[/eth/v1/validator/payload_attestation_data](https://ethereum.github.io/beacon-APIs/?urls.primaryName=dev#/Validator/producePayloadAttestationData)
| `PayloadAttestationData` |

**`attestationFreshnessOptions`, `syncCommitteeFreshnessOptions` and
`payloadAttestationFreshnessOptions`** use `WithRace`, an accept
criterion, `WithDeadline`, and `WithRepoll(UntilAccepted)`.

- `WithRace` queries all beacon nodes concurrently.
- The accept criterion (`WithAccept` for the JSON reads, `WithSSZAccept`
for the SSZ payload-attestation read) prioritizes any response whose
root matches the expected head root. If the deadline is reached without
a matching response, the validator client falls back to the first
response received. (Voting for a stale block is preferable to not voting
at all.)
- `WithDeadline` works together with the accept criterion to decide when
to use the fallback response. It is floored to `readFreshnessBudget` so
that a lagging node still gets time to import the announced head.
- `WithRepoll(UntilAccepted)` continuously re-queries all beacon nodes
until either a matching response is received or the deadline is reached.

**`blockFreshnessOptions`** uses `WithRace`, `WithSSZAccept`,
`WithRepoll(UntilAny2xx)`, and `WithDeadline`.

- `WithRace` queries all beacon nodes concurrently.
- `WithSSZAccept` prioritizes the block that is *built on top of* the
expected head, i.e. the block whose **parent** root matches the expected
head root.
- `WithRepoll(UntilAny2xx)` keeps re-polling until at least one node
returns a block (any 2xx).
- `WithDeadline` is the caller's context deadline (the slot deadline).

**The head tracker**
The freshness options above all steer requests toward the node that
already imported the *expected head*. Something has to decide what that
expected head is. This is the head tracker.

The head tracker records the latest head (block root and its slot) the
validator client has learned about from the head events of *all*
connected beacon nodes (fed by the multi-event stream). On each head
event, it keeps the head with the highest slot and ignores any event for
an older slot. So the "expected head" is the most advanced head
announced by any single beacon node.

When the validator client is about to attest, participate in a sync
committee, propose a block or attest to a payload, it attaches the
tracked head (the expected head root, plus a deadline) to the request
context as a freshness *hint*. The freshness options then read that hint
to build their accept criteria.

This is what closes the loop and solves the problem described at the
top: even if a node is lagging, a head event from any other node
advances the head tracker, so the validator client knows which head to
expect and can request fresh data from whichever node already imported
it.

```
head events (all nodes) --> head tracker (best head) --> hint on ctx --> freshness options
```

### To test
Use a validator client with the `--enable-beacon-rest-api` flag and more
than one beacon node:
```
--beacon-rest-api-provider=http://beacon-1,http://beacon-2,http://beacon-3
```

Test by powering down all beacon nodes except 1 (regardless the chosen
beacon node is).

Also, the interesting case is the following: If, amongst all the
connected beacon node, at least one imported the block before the 4
seconds deadline, then all the votes (head, source, target) should be
correct.

To ensure the correct behavior of all configurations, I tested with 3
VCs at the same time
- 1 REST VC connected in active-mode to 2 BNs
- 1 gRPC VC
- 1 REST VC connected to a single BN

Results from https://github.com/nalepae/validators-effectiveness are
displayed below:
**REST VC connected in active-mode to 2 BNs**
<img width="1023" height="1444" alt="image"
src="https://github.com/user-attachments/assets/068f4911-1924-4ff9-82c3-4ba22c78a713"
/>

**1 gRPC VC**
<img width="1030" height="1450" alt="image"
src="https://github.com/user-attachments/assets/5f71e7a1-3d32-40dc-bf49-8aea672a5488"
/>

**1 REST VC connected to a single BN**
<img width="1048" height="1448" alt="image"
src="https://github.com/user-attachments/assets/ecc938a3-887d-4134-8480-4dd67788d5d0"
/>

> [!NOTE]
> The 4 `not submitted` attestations are due to the "migrate to cold"
process locking the DB for a long time, unrelated to the current PR.

**Other notes for review**
> [!NOTE]
> Please read commit by commit, with commit messages.
> Only the REST validator client is impacted. No change on gRPC.

> [!IMPORTANT]
> As the `--enable-beacon-rest-api` flag is still in the experimental
mode, we chose to replace the current active-passive connection scheme
by the new proposed active-active connection scheme.

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
Verify weak subjectivity checkpoint against the finalized canonical chain (#17295)

- VerifyWeakSubjectivity accepted any stored block in the epoch's slot
range, so a non-canonical fork block satisfied the check. Now also
requires the root to be in the finalized block roots index.
- Verification waits until finality passes the checkpoint epoch, since
the index only becomes canonical-exact for epochs older than the latest
finalized epoch.
- Fixed the search range overshooting into the first slot of the next
epoch.
Address James' review: soft fail when initial request, when polling is enabled

Address James' review: document the behavior + log url when failed

Merge remote-tracking branch 'upstream/develop' into feat/web3signer-hot-key-reload

Remove unused reference counter from validator map handler (#17286)

The refcount on the validator pubkey to index map is never read,
`Refs()` and `MinusRef()` have no callers on it and `finalizerCleanup`
never decrements it. It has been dead since #8860 removed the
copy-on-append and #13954 removed the `Copy()` method.
Only accept gossip bids compatible with the head view (#17288)

- Dedup bids per builder on the tuple (slot, parent_block_hash,
parent_block_root) instead of per slot
- Replace the known-parent-root gossip check with
is_bid_compatible_with_head, accepting only bids that build on the head
block (full or empty variant per should_build_on_full) or on the head's
parent block
- Spec reference: https://github.com/ethereum/consensus-specs/pull/5497
Reorg late blocks even on slot 31 (#17257)

The lookahead solved the shuffling stability problem on Fulu

This PR also adds a regression test and fixes the tests that were
incorrectly passing before this PR, in fact the epoch boundary clause
was not tested.
add attestationGloas and replace usages (#17258)

Add `AttestationGloas` and replace usages throughout the codebase, in
order for codegen to be able to distinguish between the progressive and
legacy merkleization.
Fix incompatibilities between `--beacon-db-pruning` and `--enable-state-diff`. (#17287)

**What type of PR is this?**
Bug fix

**What does this PR do? Why is it needed?**
This PR has two commits.
1. The first commit fixes
https://github.com/OffchainLabs/prysm/issues/17148
2. The second commit fixes a bug hidden by
https://github.com/OffchainLabs/prysm/issues/17148: If, after
https://github.com/OffchainLabs/prysm/issues/17148 is fixed, a beacon
node both using `--beacon-db-pruning` and
`--enable-state-diff` is rebooted after the checkpoint block is itself
pruned (so, after ~5 months), then the node cannot restart.

**Which issue(s) does this PR fix?**
- https://github.com/OffchainLabs/prysm/issues/17148

**Other notes for review**
Please read commit by commit, with commit message.

**How to test**
To reproduce the bug fixed by the first commit:
1. Run the node from a fresh DB with the `--beacon-db-pruning` and
`--enable-state-diff` flags
2. Wait for the first pruning action to run (up to one epoch after the
sync)
3. Reboot the node

To reproduce the bug fixed by the second commit:
1. Checkout to the first commit (else, the bug fixed by the fixed by the
first commit will hit first)
2. Run the node from a fresh DB with the `--beacon-db-pruning` and
`--enable-state-diff` flags
3. Wait for `MIN_EPOCHS_FOR_BLOCK_REQUESTS` (~5 months)
4. Reboot the node

Note: If you are in a hurry, you can cherry pick
9af4973fd970f09e02e1c7d72c9303270627848f, then start the node with
`--pruner-retention-epochs=3` and then wait only for 3 epochs before
rebooting

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).
Fix unfilled format verb in `data_column_sidecar_` metric (#17254)

**What type of PR is this?**

> Bug fix

**What does this PR do? Why is it needed?**

> Also it seems like `p2p_topic_peer_count` is broken for data columns -
looks like a templating bug in the metric because I see one of the
values for the topic dimensions is
`/eth2/8c9f62fe/data_column_sidecar_%!d(MISSING)/ssz_snappy`

Excerpt from the Slack discussion. This PR collects
`data_column_sidecar` p2p metrics before entering `p2p.AllTopics()`
loop. Regression test added
(`TestUpdateMetrics_TopicLabelsFullyFormatted`).

**Which issue(s) does this PR fix?**

N/A

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).
Address James' review: simplify flag name

Merge remote-tracking branch 'upstream/develop' into feat/web3signer-hot-key-reload

Endpoints validator logs cleanup (#17284)

**What type of PR is this?**

 Other

**What does this PR do? Why is it needed?**

log cleanup

**Which issue(s) does this PR fix?**

Fixes #

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).
Filter sync committee contributions by the aggregator's own voted root (#17277)

- `GetSyncCommitteeContribution` filtered pool messages by
`HeadFetcher.HeadRoot()` read at aggregation time, but the messages were
signed against head at the sync message deadline earlier in the slot.
- Edge case: a block arriving after the sync message deadline moves head
between those two moments, so every pooled message carries the parent
root while the filter asks for the new block root, matching nothing and
producing an empty contribution.
- Now uses the root from the aggregator's own message in the pool,
falling back to head when that message is absent. Identical to current
behavior
Fix `TestSlashValidator_OK` flakiness. (#17279)

**What type of PR is this?**
Other

**What does this PR do? Why is it needed?**
Fix `TestSlashValidator_OK` flakiness.

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).
Enforce Web3Signer user to set flags correctly when startup (#17226)

**What type of PR is this?**

> Other: UX improvement

**What does this PR do? Why is it needed?**

Return early when building a config for Web3Signer (`Web3SignerConfig`)
when the conditions are not met. This PR also includes a small refactor
that increases the readability of the code.

**Which issue(s) does this PR fix?**

N/A

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Construct Web3Signer Keymanager directly from its config instead of temporary, fake in-memory wallet (#17219)

**What type of PR is this?**

> Other: Cleaning up our tech debt.

**What does this PR do? Why is it needed?**

This PR resolves one of our tech debt: as Prysm wallet is one and only
way to wake up VC, we create a fake Prysm wallet even for Web3Signer
(remote signer) path. This PR decouples wallet path and web3signer path
so that we can revamp our key loading process for VC.

**Which issue(s) does this PR fix?**

Part of
- #17165
- **Web3Signer keymanager without a wallet**

**Other notes for review**

Tested with our own E2E as well as Kurtosis config.

Running our E2E:
```bash
bazel test //testing/endtoend:go_minimal_scenario_test \
    --test_filter=TestEndToEnd_MinimalConfig_Web3Signer \
    --test_output=streamed
```

Note that Java 21 should be installed locally.

Running Kurtosis:
```yaml
participants:
  - el_type: geth
    cl_type: prysm
    cl_image: prysm-bn-custom-image:latest
    supernode: true
    vc_type: prysm
    vc_image: prysm-vc-custom-image:latest
    count: 2
    use_remote_signer: true

network_params:
  fulu_fork_epoch: 0
  genesis_delay: 40

additional_services:
  - dora

global_log_level: debug
```

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Fix validator monitor dropping aggregated performance logs (#17272)

**What type of PR is this?**
Bug fix

**What does this PR do? Why is it needed?**
Before this commit, `logAggregatedPerformance` ranged over a map and
used `break` to skip validators with no recorded data, so a randomized
iteration order aborted the loop at the first empty entry.
A single tracked validator without included attestations made each
5-epoch report log an arbitrary subset of monitored validators,
sometimes none. Use `continue` instead.

Also fixes the `TestLogAggregatedPerformance flake`, which passed only
when the one populated fixture entry was visited first.

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).
VC: Remove unused `FeeRecipientByPubKey` method (#17263)

**What type of PR is this?**

> Other: Cleanup

**What does this PR do? Why is it needed?**

The beacon node's GetFeeRecipientByPubKey endpoint and the proto
definitions stay until the whole gRPC service is removed.

The proposer flow pushes fee recipients to the beacon node via
PrepareBeaconProposer and never pulls them back, so this method had zero
callers. Drop the interface declaration, the beacon-api stub (which
returned nil, nil), the grpc-api implementation, the generated mock, and
the dead test scaffolding that fed it.

**Which issue(s) does this PR fix?**

N/A

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
VC: Remove unused `ValidatorParticipation` and `ValidatorQueue` from validator `ChainClient` (#17264)

**What type of PR is this?**

> Other: Cleanup

**What does this PR do? Why is it needed?**

Neither method had a caller anywhere in `validator/` — the only
references were the gRPC-fallback delegation lines themselves. Drop them
from the `iface.ChainClient` interface, the gRPC and beacon-API
implementations, and the generated mock.

The beacon node keeps serving both endpoints for external consumers.

**Which issue(s) does this PR fix?**

N/A

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
E2E: Remove `StreamBlocksAltair` which is only used in e2e (#17267)

**What type of PR is this?**

> Other: Cleanup

**What does this PR do? Why is it needed?**

`StreamBlocksAltair` is **only** used to check whether the fork happens
in our e2e tests, which is an overkill and sort of tech debt that we
have. This PR replaces streaming function by polling it every slot
second which is much easier to understand and maintain.

Removing `StreamBlocksAltair` brings a lot: we now don't have to
maintain the proto helpers and block converters. I presume nobody uses
this API as we don't have gRPC gateway at this moment.

**Which issue(s) does this PR fix?**

N/A - but will help #15346

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Add attributes event when sending on late blocks. (#17262)
updating changelog for v7.1.8 (#17252)

**What type of PR is this?**

Documentation

**What does this PR do? Why is it needed?**

Updates changelog for the v7.1.8 release.

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
Log the underlying error in the "genesis provider failed" warning (#17259)

**What type of PR is this?**

Other

**What does this PR do? Why is it needed?**

Include the dropped provider error in the "genesis provider failed"
warning log, so failures like a misconfigured checkpoint sync URL are
visible instead of only surfacing later as a generic "genesis state has
not been initialized" error.
Drop the unused error return from the beacon-chain sync `currentForkDigest` helper and its callers (#17253)

**What type of PR is this?**

> Other

**What does this PR do? Why is it needed?**

From https://github.com/OffchainLabs/prysm/pull/15490,
`currentForkDigest` never returns non-nil `err`. This PR changes
`currentForkDigest` signature only to return the digest, and fixes the
caller side by removing error handling parts.

**Which issue(s) does this PR fix?**

N/A

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).
builders progressive (#16896)

previously missed builders root progressive merklization
builder pending withdrawals progressive ssz (#16887)

previously missed Builder pending withdrawals field converted to use
progressive merklization
progressive withdrawal root and light client changes (#16885)

the `PayloadExpectedWithdrawals` was missed before.

also small changes for light client.
Don't downscore honest peers for foreign data column sidecars (#17231)

The data-column fetch flow intends to fetch columns for specific local
blocks, but `sendDataColumnSidecarsRequest` prefers a **by-range
(by-slot)** request. A slot is not a root: on a fork the node holds
orphan block `R_orphan` at slot `S`, and a peer correctly answers the
by-range request with the **canonical** block's sidecars (`R_canon`) for
that slot.

Gloas column sidecars carry a self-declared `beacon_block_root` with no
header binding to verify against. `verifyByRootDataColumnSidecars` then
looks up `blockByRoot[R_canon]`, misses (we only hold `R_orphan`), and
returns `no local block for sidecar root …`.
`verifyDataColumnSidecarsByPeer` treats that as a peer fault and
increments `BadResponsesScorer` — **unconditionally**, unlike the
RPC-layer faults. Repeated across every canonical peer, the node
disconnects them all and stays isolated on the dead fork.

## Fix

- **Drop, don't downscore.** `verifyByRootDataColumnSidecars` filters
out sidecars whose root isn't a local block *before* verification. A
by-range response for a slot where our local block differs is not a peer
fault, so it's silently dropped instead of counting as a bad response.
Genuine crypto/format faults (KZG, inclusion proof, Fulu
header-signature mismatch) still downscore the actually-faulty peer.
- **Request by root on recovery paths.** New
`DataColumnSidecarsParams.RequestByRoot` forces by-root requests (the
by-root RPC layer already validates returned roots against the request).
Set on the pending/by-root recovery path
(`rpc_beacon_blocks_by_root.go`) and the origin backfill
(`initial-sync/service.go`). Forward round-robin initial-sync keeps
by-range for throughput; the drop-not-downscore filter is its safety
net.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Support SSZ request bodies for `POST /eth/v2/beacon/pool/attestations` post-Electra (#17234)

**What type of PR is this?**

> Feature

**What does this PR do? Why is it needed?**

Add `application/octet-stream` decoding to `POST
/eth/v2/beacon/pool/attestations` for Electra+ (fixed-size
`SingleAttestation` list), matching the block-submission SSZ convention.
See
[beacon-APIs](https://ethereum.github.io/beacon-APIs/#/Beacon/submitPoolAttestationsV2).

Pre-Electra keeps the JSON path, as pre-Electra Attestation is
variable-length container thus the list of Attestation is
variable-length, which makes our code much harder to navigate. This PR
presumes that we only cares post-Electra.

**Which issue(s) does this PR fix?**

N/A - but related to #17183.

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Don't block gloas block import on missing data columns (#17232)

- Fix pending queue gloas block import when custody columns were
missing, and the pending queue fetching all 128 columns every ~4s.
Orphaned or unrevealed payloads have no columns anywhere,
- `requestAndSaveMissingDataColumnSidecars` now drains queued gossip
columns for gloas blocks and skips the peer fetch. Gloas DA is checked
on the payload envelope, not at block import already today
progressive hashTreeRoot (#16861)

progressive merklization of beacon state container.

note:
this implementation is naive and it's ignoring all the partial rebuilds.
the goal is to have a correct implementation and then optimize it.

---------

Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
`ProgressiveList`: Chunk directly byte slice instead of `PackByChunk` (#17209)

**What type of PR is this?**

> Other: Optimization

**What does this PR do? Why is it needed?**

This PR chunks a byte slice directly when it is given for calculating
the HTR. Previously, `PackByChunk` is used but chunking it directly can
be used for memory optimization. This can save up to 5MB for each
`*_participation` fields in `BeaconState` when we have 1M validators,
which is close to the current number in mainnet.

**Which issue(s) does this PR fix?**

N/A

**Other notes for review**

Running benchmark:

in `progressive_byteslice_bench_test.go`

```go
func BenchmarkByteSliceRootProgressive(b *testing.B) {
	for _, n := range []int{262144, 1048576} {
		slice := make([]byte, n)
		for i := range slice {
			slice[i] = 0b111
		}
		b.Run(fmt.Sprintf("n=%d", n), func(b *testing.B) {
			b.ReportAllocs()
			for b.Loop() {
				if _, err := ssz.ByteSliceRootProgressive(slice); err != nil {
					b.Fatal(err)
				}
			}
		})
	}
}
```

with this command:

```bash
bazel test //encoding/ssz:go_default_test \
    --test_filter='^$' \
    --test_arg=-test.bench=BenchmarkByteSliceRootProgressive/n=1048576 \
    --test_arg=-test.benchtime=2s \
    --test_arg=-test.count=1 \
    --test_arg=-test.benchmem \
    --test_output=streamed \
    --nocache_test_results
```

### Result

#### Before

```
goos: darwin
goarch: arm64
cpu: Apple M5 Pro
BenchmarkByteSliceRootProgressive/n=1048576-18              1981           1095825 ns/op         7703541 B/op        340 allocs/op
```

#### After
```
goos: darwin
goarch: arm64
cpu: Apple M5 Pro
BenchmarkByteSliceRootProgressive/n=1048576-18              3032            807473 ns/op         2768350 B/op        315 allocs/op
```

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).
update go toolchain to 1.26.5 (#17230)

**What type of PR is this?**
Other

**What does this PR do? Why is it needed?**
Golang released a patch level update on 7/7 with various fixes
([changelog](https://go.dev/doc/devel/release#go1.26.5)). This updates
prysm to use the latest patches.

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
Self code review: `equalKeySet` had assumed deduped key set but make it strict so not relying on the naive assumption.

Self code review: Serialize `updatePublicKeys` with mutex.

Now we have three concurrent updaters of the key set: the URL poller, the file watcher, and the Keymanager API.
Previously, we only used a read-write lock to guard access to the key set,
but this does not guarantee that the order of events in the `accountsChangedFeed` matches the order of committed key sets.

Only count actually invalid envelopes in envelope invalid metric (#17215)

- `beacon_execution_payload_envelope_invalid_total` incremented on any
`ReceiveExecutionPayloadEnvelope` error, including local failures like
missing prestate or EL timeouts, so a flaky EL reads as invalid
envelopes
- Count only EL INVALID status and consensus verification failures
Add changelog

Implement hot reload for public keys in Web3Signer URL path

Add `DecodeHex48` for helper

Add `--validators-external-signer-key-poll-interval` flag to allow hot-reloading of public keys from a remote web3signer URL.

Zero means no polling. Consider no-op when no public keys URL is provided.

Move file watcher logics into separate file (`key_source_file.go`)

No functional behavior changes. Includes unit tests.

wire progressive ssz functions into state fields (#16860)

change the merklization of related state fields to progressive.

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Fix PTC blob_data_available to use data column availability (#17222)

- `BlobDataAvailable` in payload attestation data was populated from
`HasFullNode`, which only flips after the envelope is fully imported
- Per spec, `blob_data_available` is
`is_data_available(beacon_block_root)`, independent of envelope import
- New `DataAvailable` getter checks the non-blocking column store status
first, and only reads the block's bid when no columns are stored, since
an empty column summary can't distinguish a blobless payload from
missing data and this order avoids a DB block read in the common case
Validate bid parent block hash correctly (#17217)

- Bid validation resolved the expected parent hash via
`ForkChoice.BlockHash`, which returns the hash committed in the parent
block's bid even when the payload was never revealed.
- Add `ForkChoice.HasPayloadBlockHash(root, hash)`
- Change `VerifyParentBlockHash` to take a `(root, hash) bool` lookup
and wire the new method into bid gossip validation and
`SubmitSignedExecutionPayloadBid`
add ssz functions for progressive merkliezation (#16847)

Adding the progressive merkliezation functions for the SSZ package.

- `MerkleizeProgressiveChunks`
- `MerkleizeVectorSSZProgressive`
- `MerkleizeListSSZProgressive`
- `SliceRootProgressive`
- `ByteSliceRootProgressive`
- `MixInActiveFields`
Remove unused data_column_obtained_via_el_count metric (#17216)

- Defined but never recorded anywhere, permanently zero
- `data_columns_recovered_from_el_{attempts,total}` already cover this
Increment newPayload node count metrics on Gloas envelope path (#17213)

- `new_payload_{valid,optimistic,invalid}_node_count` were only
incremented on the pre-Gloas block path, so they stay flat under Gloas
while the envelope path makes all the newPayload calls
- Increment them in `callNewPayload`
e2e: Update lighthouse version in multiclient testing (#17221)

**What type of PR is this?**

Other

**What does this PR do? Why is it needed?**

This updates lighthouse version used in multiclient e2e tests. This
comment is taken from PR #17134 with all credits to @nalepae.

**Which issue(s) does this PR fix?**

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
Only increment PTC vote count metric for newly set votes (#17214)

- `forkchoice_ptc_vote_count` incremented on every `SetPTCVote` call, so
the same vote applied from gossip and again from a block aggregate
counted twice (~883/slot observed on devnet-7 vs PTC size 512)
- Only increment when the attester bit was previously unset
fix(p2p): preserve valid static peers (#17192)

<!-- Thanks for sending a PR! Before submitting:

1. If this is your first PR, check out our contribution guide here
https://docs.prylabs.network/docs/contribute/contribution-guidelines
You will then need to sign our Contributor License Agreement (CLA),
which will show up as a comment from a bot in this pull request after
you open it. We cannot review code without a signed CLA.
2. Please file an associated tracking issue if this pull request is
non-trivial and requires context for our team to understand. All
features and most bug fixes should have
an associated issue with a design discussed and decided upon. Small bug
   fixes and documentation improvements don't need issues.
3. New features and bug fixes must have tests. Documentation may need to
be updated. If you're unsure what to update, send the PR, and we'll
discuss
   in review.
4. Note that PRs updating dependencies and new Go versions are not
accepted.
   Please file an issue instead.
5. A changelog entry is required for user facing issues.
-->

**What type of PR is this?**

> Bug fix

**What does this PR do? Why is it needed?**

`PeersFromStringAddrs` previously returned immediately when an ENR could
be parsed but could not be converted into a dialable multiaddress. As a
result, one unusable static peer caused all other valid addresses in the
same peer list to be discarded.

This change logs the per-peer conversion error and continues processing
the remaining entries. It also adds a regression test covering a list
containing both a valid multiaddress and an unusable ENR.

**Which issue(s) does this PR fix?**

Fixes #17115

**Other notes for review**

Testing:

- `bazel test //beacon-chain/p2p:go_default_test
--test_filter=TestPeersFromStringAddrs_SkipsUnusablePeer`
- `make build`

The full P2P test target was also attempted. The existing
network-dependent
`TestService_BroadcastAttestationWithDiscoveryAttempts` test timed out
after five minutes on all three retries; the targeted regression test
passes.

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).
Mark pending payload envelope seen only after successful import (#17206)

- Pending payload envelopes were marked seen before
`ReceiveExecutionPayloadEnvelope`, so a transient import failure
poisoned the seen cache and blocked retries via gossip and by-root
refetch.
- Mark seen only after successful import, keeping the bad-block path
marked seen.
making duties failures independent of each other (#17036)

**What type of PR is this?**
 Feature

**What does this PR do? Why is it needed?**

#16421 split validator-client duty fetching across gRPC/REST and
introduced **promotion** — instead of re-pulling both current and next
epoch every boundary, the cached next-epoch duties are promoted into the
current epoch and only the new next-epoch is fetched.

This PR finishes the two follow-ups #16928 left open:

1. **Next-epoch duties are always optional.** Previously a next-epoch
*attester* fetch failure was a hard error that aborted the whole update
— disrupting the *current* epoch too. Now every next-epoch duty type
fails softly (flagged, not fatal), so a next-epoch problem never breaks
the current epoch.
2. **Failures are retried per-type, not bundled.** Previously any single
missing next-epoch duty forced a full re-pull of *all four* duty types
at the next boundary. Now the missing types are re-fetched
**individually, mid-epoch**, and merged in — so promotion stays cheap
and a single transient failure doesn't cascade.

A third, related change keeps the dependent-root (reorg) handling
consistent with the new soft-failure model.

## What changed

### Next-epoch duties optional
`missingNextDuties` gains a `missingNextAttester` bit. In both the
full-fetch and promotion paths, a failed next-epoch attester is logged
and flagged (not returned as an error), exactly like proposer/sync/ptc
already were. Current-epoch attester/proposer remain required.

### Per-type mid-epoch retry
- `RetryMissingNextDuties` re-fetches **only** the duty types flagged
missing and overlays them onto the existing next-epoch duties
(`overlayNextDuties`), leaving the types that already succeeded
untouched. If the **attester spine** itself is missing (the attester
assignment is what creates each validator's duty row, so without it
there are no rows to overlay onto), it rebuilds the whole next epoch
instead.
- `MaybeRetryMissingNextDuties` is what the run loop calls each
non-epoch-start slot. It runs the retry in its **own goroutine** so the
current slot's attestation/proposal isn't blocked, but only when there's
actually missing work (`needsNextRetry`) and no retry is already in
flight (`retryInFlight` CAS) — so we don't spawn goroutines for nothing
or pile them up.
- Writes are guarded by a store **revision** counter: if an
epoch-boundary or head-event update lands while a retry goroutine is
mid-fetch, the now-stale write is dropped (`replaceNextDuties` applies
only if the store is still at the revision the fetch was based on).

### Dependent-root / reorg handling
`checkDependentRoots` no longer treats an *unknown* (nil) current
dependent root as "needs update" — that state only arises after a soft
next-epoch attester failure, and forcing a full `UpdateDuties` on every
head event while it's nil is wasteful. Recovery is instead owned by the
epoch boundary and the per-slot retry. Current-epoch reorg detection
(the `prev` dependent-root path) is unchanged.

## Before / after

### Per-duty-type fetch failure

| Duty type | Current epoch fails | Next epoch fails — **before** | Next
epoch fails — **now** |
|---|---|---|---|
| **Attester** | hard error → keep cached duties, retry next tick |
**hard error → aborts the whole update, breaking the current epoch** |
soft: flag `missingNextAttester`; spine rebuilt by the mid-epoch retry |
| **Proposer** | hard error → keep cache | soft, but forced a full
4-type re-pull next boundary | soft: flag; overlaid by the mid-epoch
retry |
| **Sync** | soft (logged) | soft, forced a full 4-type re-pull | soft:
flag; overlaid by the mid-epoch retry |
| **PTC** | soft (logged) | soft, forced a full 4-type re-pull | soft:
flag; overlaid by the mid-epoch retry |

### Triggers → action

| Trigger | **Before** | **Now** |
|---|---|---|
| Epoch boundary, no failures + same validators + stable dep root |
promote next→current, fetch next epoch only | same |
| Epoch boundary, can't promote (first run / validator-set drift) | full
fetch (current + next, all types) | same |
| Epoch boundary, a next-epoch type failed last cycle | **forced full
re-pull of all 4 types** | **promote** — the gap was already filled
mid-epoch |
| Mid-epoch slot | *(nothing — wait for next boundary)* | **retry only
the missing next-epoch types**, in a goroutine off the slot critical
path; no-op when nothing is missing |
| Head event, dependent root changed | `UpdateDuties` | same |
| Head event, current dependent root **unknown (nil)** | **trigger
`UpdateDuties`** | **skip** — let the boundary + per-slot retry recover
|

**Which issue(s) does this PR fix?**

Fixes #https://github.com/OffchainLabs/prysm/issues/16928

**Other notes for review**

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
`--max-builder-consecutive-missed-slots` and `--max-builder-epoch-missed-slots` help text: Reports the mainnet config value as the default instead of a hardcoded, false and unused values. (#17211)

**What type of PR is this?**
Bug fix

**What does this PR do? Why is it needed?**
Currently, this is the printed help:

```
--max-builder-consecutive-missed-slots value  Number of consecutive skip slot to fallback from using relay/builder to local execution engine for block construction (default: 3)
--max-builder-epoch-missed-slots value        Number of total skip slot to fallback from using relay/builder to local execution engine for block construction in last epoch rolling window. The values are on the basis of the networks and the default value for mainnet is 5. (default: 0)
```

There is 2 issues here:
1. The displayed default value of
`--max-builder-consecutive-missed-slots` is the correct one, but is
actually hard coded. The real default value comes from the config.
2. The displayed default value of `--max-builder-epoch-missed-slots` is
simply wrong.

This PR fixes these 2 issues, by using the correct default values.

Now:
```
 --max-builder-consecutive-missed-slots value  Number of consecutive skip slot to fallback from using relay/builder to local execution engine for block construction (default: 3)
 --max-builder-epoch-missed-slots value …
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants