Skip to content

Introduce AttestationMatchingGroupV2#9438

Merged
tbenr merged 12 commits intoConsensys:masterfrom
tbenr:attestation-matching-groupV2
May 20, 2025
Merged

Introduce AttestationMatchingGroupV2#9438
tbenr merged 12 commits intoConsensys:masterfrom
tbenr:attestation-matching-groupV2

Conversation

@tbenr
Copy link
Copy Markdown
Contributor

@tbenr tbenr commented May 19, 2025 •
edited
Loading

The new matching group has:

  • concurrency friendly by removing synchronized methods
  • internal separation among aggregates and single attestations
  • clear separation between flows:
    • streamForBlockProduction - consider only aggregates
    • streamForAggregationProduction - consider only single attestations (for electra)
    • streamForApiRequest - consider all attestations (aggregates and single)
  • an additional fillUp step which will be used by the pool v2 to improve aggregates produced by the streamForBlockProduction flow.
  • timelimit on aggregation, which is used streamForAggregationProduction and fillUp steps. This allows to timebox the work and have a good guarantee we complete aggregation in a reasonable time no matter what is in the pool
  • a configuration param that enables "early single attestations drop" to discard SA as aggregates come in the pool

Documentation

  • I thought about documentation and added the doc-change-required label to this PR if updates are required.

Changelog

  • I thought about adding a changelog entry, and added one if I deemed necessary.

tbenr
tbenr commented May 19, 2025
View reviewed changes
Comment thread ...rc/main/java/tech/pegasys/teku/statetransition/attestation/utils/AttestationBitsElectra.java
}

@Override
public boolean isFromCommittee(final int committeeIndex) {
Copy link
Copy Markdown
Contributor Author

@tbenr tbenr May 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I forgot to add tests for these new methods

rolfyone
rolfyone reviewed May 20, 2025
View reviewed changes
Comment thread .../main/java/tech/pegasys/teku/statetransition/attestation/MatchingDataAttestationGroupV2.java Outdated
candidate -> includedValidators.isSuperSetOf(candidate.bits())));

// Also remove empty sets from the outer map for cleanup
attestationsByValidatorCount.entrySet().removeIf(entry -> entry.getValue().isEmpty());
Copy link
Copy Markdown
Contributor

@rolfyone rolfyone May 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

potentially we could do this after the cleanup from singleAttestationsByCommitteeIndex

This may be helpful because we'd be able to finish with included validators and release lock earlier potentially (before we do the math or this entryset we could release writelock)

Copy link
Copy Markdown
Contributor Author

@tbenr tbenr May 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking at that code section I think I can make attestationsByValidatorCount handling more like singleAttestationsByCommitteeIndex where the removal is all done in one pass

Copy link
Copy Markdown
Contributor Author

@tbenr tbenr May 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah no, that's not a good idea

Copy link
Copy Markdown
Contributor Author

@tbenr tbenr May 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes i found a good solution

rolfyone
rolfyone approved these changes May 20, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@rolfyone rolfyone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tbenr tbenr merged commit 0f7c19f into Consensys:master May 20, 2025
15 of 16 checks passed
@tbenr tbenr deleted the attestation-matching-groupV2 branch May 20, 2025 09:48
zilm13 added a commit that referenced this pull request May 20, 2025
@zilm13 @StefanBratanov
@lucassaldanha @rolfyone @tbenr @crStiv @mehdi-aouadi @gfukushima
Peerdas + master upstream + test fixes (#9436)
4422a9b 
* Add read functionality to blobs archiving (#9318)

* Schedule Gnosis Pectra hard-fork (#9340)

* Updated ref tests v1.5.0-beta.5 (#9341)

* Created Fulu skeleton (#9325)

* Made checkpoint-state-url and initial-state mutually exclusive (#9342)

It's hard to reason on how these two should interact, and logically they're mutually exclusive, so this just solidifies the theory that they shouldn't be used together.

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* partial 3rd party updates and errorprone updates (#9351)

I ended up disabling the instanceof errorprone, as it's very very common for us to be doing it apparently. I started changing things but in the interest of time will raise a ticket

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Make builder timeouts only for the HTTP call (#9353)

Co-authored-by: Lucas Saldanha <lucas.saldanha@consensys.net>

* more 3rd party updates (#9355)


Signed-off-by: Paul Harris <paul.harris@consensys.net>

* KZG updates from das branch (#9335)

* Avoid combining validators by aggregating committee when using DVT (#9357)

* builder json format, make media type more compatible (#9360)

* refactored expected withdrawals and added test cases (#9361)

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Use `URI.create(...).toURL()` removing URL constructor deprecation (#9364)

* clear-changelog (#9366)

* New infra stream module (#9362)

* feat: Add support for reading static peers from file (#9328)

Co-authored-by: Paul Harris <paul.harris@consensys.net>

* Start cleaning up `api/schema` (#9376)

* Start cleaning up `api/schema`

 - moved `ValidatorStatus`
 - Removed `Version`
 - Moved `PublicKeyException`
 - Moved `EventType`

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* cleanup the rest of api/schema (#9377)

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Added KZG computeCells method (#9375)

* Added KZG computeCells method
* Moved KZG no-operation logic into testFixture class

* Fulu Schemas (ssz + datastructures) - including fulu ssz ref tests (#9363)

* Added context to the sync committee duties error (#9380)

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Returning custody_group_count on node/identity Beacon API (#9381)

* may 3rd party updates (#9388)

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Added snakeyaml to allow more complicated config reading (#9390)

 - added tests to show reading lists of objects (BPO related)
 - added a testcase to show that hex is read correctly
 - added testcase to show specConfigReader isnt filling defaults when we read local config files

 Partially addresses #9365

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Add FULU to the Spec Factory (#9373)

* Improve attestation bits aggregator electra (#9393)

* added a DeserializableConfigTypeDefinition (#9396)

- Tests demonstrate that this should at least be very close to what we require for reading BPO schedule.

 Partially addresses #9365

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Updating ref test to 1.5.0 stable (#9398)

* uniforms the AggregatingAttestationPool interface (#9401)

* Avoids potential mutability of the result of getCommitteeBits (#9402)

* Adding DataColumnsByRootIdentifier container (#9399)

* Added an info message for the highest milestone (#9405)

Also added a sanity check that will fail if the specFactory hasn't defined the highest milestone correctly, which has flow on effects.

fixes #9400

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Introduce `PooledAttestation` and `PooledAttestationWithData` (#9407)

* Improve jdk 24 support and add docker-jdk24 (#9410)

* add docker-jdk24 support

* solves jdk 24 startup warning

* rename AttestationBitsAggregator to AttestationBits (#9412)

* Introduce aggregating pool interface (#9414)

* [docker-jdk24] SIGHUP handler fix (#9413)

* Optionally include validator indices in `PooledAttestation` (#9418)

* Improve committeesSize retrieval in `AggregatingAttestationPool` (#9419)

* Batch attestation duty scheduling for an epoch (#9374)

* Updated MiscHelpersFulu + Added Gossip Logger (#9422)

* Updated MiscHelpersFulu
* Added GossipLogger and DAS related changes

* gossip DAS related changes

* fix test

* fix runtime with noop fulu managers

* bump MAX_SUBSCRIBED_TOPICS for Fulu and further forks

* Use master's circle ci config

* Remove duplicated workflows

* Update test fixtures with random order changes

* fix test

* fix test

* fix GetNewBlockV3Test

* fix GetBlockTest

* Gossip DAS related changes (#9423)

* added bpo parsing to configuration (#9406)

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Fix ProduceBlockRequestTest for Fulu and updated order in random

* Regenerate test fixtures with the changes in DataStructureUtil call order

* Regenerate test fixtures with the changes in DataStructureUtil call order for GetNewBlockV3IntegrationTest

* spotless

* move GetDataColumnSidecars to tekuv1 space, fix openapi integration test

* BPO - remove max blobs from fulu spec (#9427)

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Introduces `RewardBasedAttestationSorter` (#9428)

* Fix integration tests after master upstream

* Fix property tests + refactor some random datastructure utils to avoid fulu duplication

* fix assemble

* fix test

* revert das test coverage changes

* DAS storage changes (#9432)

* Added config defaulting to the Config loader (#9439)

Signed-off-by: Paul Harris <paul.harris@consensys.net>
Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* Introduce `AttestationMatchingGroupV2` (#9438)

* Fix DasSyncAcceptanceTest, fix DSL, remove broken DSL

---------

Signed-off-by: Paul Harris <paul.harris@consensys.net>
Co-authored-by: Stefan Bratanov <stefan.bratanov93@gmail.com>
Co-authored-by: Lucas Saldanha <lucas.saldanha@consensys.net>
Co-authored-by: Paul Harris <paul.harris@consensys.net>
Co-authored-by: Enrico Del Fante <enrico.delfante@consensys.net>
Co-authored-by: crStiv <cryptostiv7@gmail.com>
Co-authored-by: Mehdi AOUADI <mehdi.aouadi@consensys.net>
Co-authored-by: Lucas Saldanha <lucascrsaldanha@gmail.com>
Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
zilm13 added a commit that referenced this pull request May 21, 2025
@zilm13 @StefanBratanov
@lucassaldanha @rolfyone @tbenr @crStiv @mehdi-aouadi @gfukushima
Merge latest master in das (#9456)
1c043ed 
* Add read functionality to blobs archiving (#9318)

* Schedule Gnosis Pectra hard-fork (#9340)

* Updated ref tests v1.5.0-beta.5 (#9341)

* Created Fulu skeleton (#9325)

* Made checkpoint-state-url and initial-state mutually exclusive (#9342)

It's hard to reason on how these two should interact, and logically they're mutually exclusive, so this just solidifies the theory that they shouldn't be used together.

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* partial 3rd party updates and errorprone updates (#9351)

I ended up disabling the instanceof errorprone, as it's very very common for us to be doing it apparently. I started changing things but in the interest of time will raise a ticket

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Make builder timeouts only for the HTTP call (#9353)

Co-authored-by: Lucas Saldanha <lucas.saldanha@consensys.net>

* more 3rd party updates (#9355)


Signed-off-by: Paul Harris <paul.harris@consensys.net>

* KZG updates from das branch (#9335)

* Avoid combining validators by aggregating committee when using DVT (#9357)

* builder json format, make media type more compatible (#9360)

* refactored expected withdrawals and added test cases (#9361)

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Use `URI.create(...).toURL()` removing URL constructor deprecation (#9364)

* clear-changelog (#9366)

* New infra stream module (#9362)

* feat: Add support for reading static peers from file (#9328)

Co-authored-by: Paul Harris <paul.harris@consensys.net>

* Start cleaning up `api/schema` (#9376)

* Start cleaning up `api/schema`

 - moved `ValidatorStatus`
 - Removed `Version`
 - Moved `PublicKeyException`
 - Moved `EventType`

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* cleanup the rest of api/schema (#9377)

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Added KZG computeCells method (#9375)

* Added KZG computeCells method
* Moved KZG no-operation logic into testFixture class

* Fulu Schemas (ssz + datastructures) - including fulu ssz ref tests (#9363)

* Added context to the sync committee duties error (#9380)

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Returning custody_group_count on node/identity Beacon API (#9381)

* may 3rd party updates (#9388)

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Added snakeyaml to allow more complicated config reading (#9390)

 - added tests to show reading lists of objects (BPO related)
 - added a testcase to show that hex is read correctly
 - added testcase to show specConfigReader isnt filling defaults when we read local config files

 Partially addresses #9365

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Add FULU to the Spec Factory (#9373)

* Improve attestation bits aggregator electra (#9393)

* added a DeserializableConfigTypeDefinition (#9396)

- Tests demonstrate that this should at least be very close to what we require for reading BPO schedule.

 Partially addresses #9365

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Updating ref test to 1.5.0 stable (#9398)

* uniforms the AggregatingAttestationPool interface (#9401)

* Avoids potential mutability of the result of getCommitteeBits (#9402)

* Adding DataColumnsByRootIdentifier container (#9399)

* Added an info message for the highest milestone (#9405)

Also added a sanity check that will fail if the specFactory hasn't defined the highest milestone correctly, which has flow on effects.

fixes #9400

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Introduce `PooledAttestation` and `PooledAttestationWithData` (#9407)

* Improve jdk 24 support and add docker-jdk24 (#9410)

* add docker-jdk24 support

* solves jdk 24 startup warning

* rename AttestationBitsAggregator to AttestationBits (#9412)

* Introduce aggregating pool interface (#9414)

* [docker-jdk24] SIGHUP handler fix (#9413)

* Optionally include validator indices in `PooledAttestation` (#9418)

* Improve committeesSize retrieval in `AggregatingAttestationPool` (#9419)

* Batch attestation duty scheduling for an epoch (#9374)

* Updated MiscHelpersFulu + Added Gossip Logger (#9422)

* Updated MiscHelpersFulu
* Added GossipLogger and DAS related changes

* Gossip DAS related changes (#9423)

* added bpo parsing to configuration (#9406)

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* BPO - remove max blobs from fulu spec (#9427)

Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Introduces `RewardBasedAttestationSorter` (#9428)

* DAS storage changes (#9432)

* Added config defaulting to the Config loader (#9439)

Signed-off-by: Paul Harris <paul.harris@consensys.net>
Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* Introduce `AttestationMatchingGroupV2` (#9438)

* Introduce `AggregatingAttestationPoolV2` (#9445)

* Introduce `AggregatingAttestationPoolProfiler` implementations (#9447)

Co-authored-by: Paul Harris <paul.harris@consensys.net>

* PeerDAS RPC related changes (#9444)

* Added a development flag to enable strict config loading (#9449)

Strict loading was the default until yesterday, but it's caused a number of problems because if we push changes too fast etc it causes support issues.

The new default permissive loading would mean that if a passed config file is missing an entry, we default it and print a warning.

This new flag `--Xstartup-strict-config-loader-enabled` defaults to false, but if turned on will revert to failing on startup if a config being loaded doesn't include all of the expected configuration attributes.

We could potentially start using this in AT to specify minimal changes to a configuration for a test, but for now I've left acceptance tests using strict loading.


Signed-off-by: Paul Harris <paul.harris@consensys.net>

* Updating reference tests to 1.6.0-alpha.0 (#9450)

* Updating reference tests to 1.6.0-alpha.0

* Added Fulu networking tests

* rename

* Refactoring networking ATs

* Add peer via api (#9431)

* add initial impl of endpoint

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* add catch to cover address conversion failure

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* add tests for 200, 400 and 500

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* spotless

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* add test for when discovery is not enabled

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* spotless

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* fix name of test after changing exception returned for invalid peer

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* add json def to paths

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* add changelog

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* PR review

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* spotless caught me again

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* change impl and tests to handle a single peer per api call

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* spotless

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

---------

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* `AggregatingAttestationPool` new CLI params (#9452)

* poolV2-CLI

* fix arity

* getBits method names clearer (#9453)

* Validate electra attestationData index received by BN (#9430)

* fix integration test

---------

Signed-off-by: Paul Harris <paul.harris@consensys.net>
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Co-authored-by: Stefan Bratanov <stefan.bratanov93@gmail.com>
Co-authored-by: Lucas Saldanha <lucas.saldanha@consensys.net>
Co-authored-by: Paul Harris <paul.harris@consensys.net>
Co-authored-by: Enrico Del Fante <enrico.delfante@consensys.net>
Co-authored-by: crStiv <cryptostiv7@gmail.com>
Co-authored-by: Mehdi AOUADI <mehdi.aouadi@consensys.net>
Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rolfyone rolfyone rolfyone approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tbenr @rolfyone