feat: Protogalaxy folding of challenges#2935
Conversation
Benchmark resultsMetrics with a significant change:
Detailed resultsAll benchmarks are run on txs on the This benchmark source data is available in JSON format on S3 here. Values are compared against data from master at commit L2 block published to L1Each column represents the number of txs on an L2 block published to L1.
L2 chain processingEach column represents the number of blocks on the L2 chain where each block has 16 txs.
Circuits statsStats on running time and I/O sizes collected for every circuit run across all benchmarks.
MiscellaneousTransaction sizes based on how many contracts are deployed in the tx.
|
maramihali
left a comment
There was a problem hiding this comment.
In my opinion, relation parameters folding should be done in the prover rather than instances because i believe it would make things more clear. But maybe I am not seeing the reasoning behind your choice of doing it prefolding.
I think the terminology partial and total combined with relation and subrelation gets sometimes confusing and I tried to highlight all the points where some clarification would be needed.
| #include "barretenberg/proof_system/relations/gen_perm_sort_relation.hpp" | ||
| #include "barretenberg/proof_system/relations/lookup_relation.hpp" | ||
| #include "barretenberg/proof_system/relations/permutation_relation.hpp" | ||
| #include "barretenberg/proof_system/relations/relation_parameters.hpp" |
There was a problem hiding this comment.
was this include necessary here, i see it's also in other bench files that actually use relation parameters
There was a problem hiding this comment.
This file does use them on Line 26.
| // MAX_RANDOM_RELATION_LENGTH = algebraic degree of sumcheck relation *after* multiplying by the `pow_zeta` random | ||
| // polynomial e.g. For \sum(x) [A(x) * B(x) + C(x)] * PowZeta(X), relation length = 2 and random relation length = 3 | ||
| static constexpr size_t MAX_RANDOM_RELATION_LENGTH = MAX_RELATION_LENGTH + 1; | ||
| // BATCHED_RELATION_PARTIAL_LENGTH = algebraic degree of sumcheck relation *after* multiplying by the `pow_zeta` |
There was a problem hiding this comment.
please explain what partial means here
There was a problem hiding this comment.
also, to be matching with the line above i'd use BATCHED_PARTIAL_RELATION_LENGTH
There was a problem hiding this comment.
But it's the "partial length of the batched relation" we're computing here, not the "batching of the partial relation length", so I think what I wrote is more meaningful.
There was a problem hiding this comment.
To avoid duplicating the same comments in every flavor file, I'll write descriptions of "total" versus "partial" in the comments that define the functions. That should be good enough, right? I think anybody who's curious would look there.
There was a problem hiding this comment.
yes, that makes sense, also did not realise the nuance until now, self note to take ordering into consideration in a name
| * @brief A container for univariates produced during the hot loop in sumcheck. | ||
| */ | ||
| using ExtendedEdges = ProverUnivariates<MAX_RELATION_LENGTH>; | ||
| using ExtendedEdges = ProverUnivariates<MAX_PARTIAL_RELATION_LENGTH>; |
There was a problem hiding this comment.
i find the usage of partial in the context of sumcheck a bit strange because in that case the partial degree is actually the total degree :-?
| /** | ||
| * @brief Create folded (univariate) relation parameters. | ||
| * @details For a given relation parameter type, extract that parameter from each instance, place the values in a | ||
| * univariate (i.e., sum them against an appropriate Lagrange basis) and then extended as needed during the |
There was a problem hiding this comment.
| * univariate (i.e., sum them against an appropriate Lagrange basis) and then extended as needed during the | |
| * univariate (i.e., sum them against an appropriate univariate Lagrange basis) and then extended as needed during the |
| auto& univariate_param = *params_to_fold[param_idx]; | ||
| Univariate<FF, NUM> tmp(0); | ||
| for (size_t instance_idx = 0; instance_idx < NUM; instance_idx++) { | ||
| tmp.value_at(instance_idx) = *((*_data[instance_idx]).relation_parameters.to_fold[param_idx]); |
There was a problem hiding this comment.
how should I feel about double dereferencing?😅
There was a problem hiding this comment.
Ah, thanks for raising this yes what I did here was no good and I thought I'd left a TODO note to help me catch it. Will rework.
| } | ||
|
|
||
| template <typename Accumulator0, size_t read_index, typename AllEntities> | ||
| static Accumulator0 compute_read_term_predicate(const AllEntities& in) |
There was a problem hiding this comment.
Becaue in practice it's extracted as the 0th type of a tuple type. Looking at it here again I thought it seemed a little odd. Can revert if you like.
barretenberg/cpp/src/barretenberg/proof_system/relations/relation_types.hpp
Outdated
Show resolved
Hide resolved
| using SumcheckTupleOfUnivariatesOverSubrelations = TupleOfUnivariates<FF, RelationImpl::SUBRELATION_LENGTHS>; | ||
| using SumcheckArrayOfValuesOverSubrelations = ArrayOfValues<FF, RelationImpl::SUBRELATION_LENGTHS>; | ||
| TupleOfUnivariates<FF, | ||
| compute_composed_SUBRELATION_PARTIAL_LENGTHS<NUM_INSTANCES>( |
| */ | ||
| template <typename RelationImpl> | ||
| consteval std::array<size_t, RelationImpl::SUBRELATION_PARTIAL_LENGTHS.size()> compute_full_subrelation_lengths() | ||
| { |
There was a problem hiding this comment.
if this is regarding challenges as variables, shouldn't this be total length or am I misunderstanding something?
There was a problem hiding this comment.
Yeah I started with "full" and then decided "total" was better but I missed the renaming here, thanks.
| * | ||
| * @todo TODO(https://github.com/AztecProtocol/barretenberg/issues/720) | ||
| * @note We use some funny terminology: we use the term "length" for 1 + the degree of a relation. When the relation is | ||
| * regarded as a polynomial in all of its arguments, we refer refer to this length as the "total length", and when we |
| * univariate (i.e., sum them against an appropriate univariate Lagrange basis) and then extended as needed during | ||
| * the constuction of the combiner. | ||
| */ | ||
| static void fold_parameters(ProverInstances& instances) |
🤖 I have created a release *beep* *boop* --- <details><summary>aztec-packages: 0.13.0</summary> ## [0.13.0](aztec-packages-v0.12.0...aztec-packages-v0.13.0) (2023-10-31) ### ⚠ BREAKING CHANGES * PXE.getNotes(...) + refactor of note types ([#3051](#3051)) ### Features * `FieldNote` ([#3037](#3037)) ([3d1ffd0](3d1ffd0)) * Add Aztec Boxes page to docs ([#2569](#2569)) ([997c15c](997c15c)) * Adding structure to Transcript ([#2937](#2937)) ([db67aa1](db67aa1)) * Compile noir contracts with noir_wasm ([#2737](#2737)) ([524cecf](524cecf)) * Dockerize aztec-cli ([#3031](#3031)) ([ec2e3c2](ec2e3c2)) * Efficient ZM quotient computation ([#3016](#3016)) ([ebda5fc](ebda5fc)) * **feature_branch:** Private Kernel Circuit ([#2740](#2740)) ([f800a36](f800a36)) * Measure plonk rounds ([#3065](#3065)) ([c8e1d8b](c8e1d8b)) * Migrate the init kernel CPP tests to noir ([#3091](#3091)) ([906429f](906429f)) * New script to output table of benchmarks for README pasting. ([#2780](#2780)) ([6c20b45](6c20b45)) * Pedersen in typescript. ([#3111](#3111)) ([933f1b2](933f1b2)) * Protogalaxy folding of challenges ([#2935](#2935)) ([7ed30e8](7ed30e8)) * PXE.getNotes(...) + refactor of note types ([#3051](#3051)) ([16abb5a](16abb5a)) * Zeromorph with concatenation (Goblin Translator part 10) ([#3006](#3006)) ([70b0f17](70b0f17)) ### Bug Fixes * Bad contract txs publishing contract data ([#2673](#2673)) ([ccd4611](ccd4611)) * Better error message for compute_note_hash_and_nullifier. ([#3097](#3097)) ([57bec53](57bec53)) * Broken `FieldNote` test ([#3135](#3135)) ([fe78ecf](fe78ecf)) * Docker-compose up, rather than run. ([#3081](#3081)) ([242f780](242f780)) * Formatting ([#3070](#3070)) ([e1633d3](e1633d3)) * Minor stale naming fix ([#3117](#3117)) ([a6786ae](a6786ae)) * Push cli docker image to docker hub ([#3120](#3120)) ([ccad50f](ccad50f)) * Remove duplicate terraform resource definition ([#3066](#3066)) ([d5abadb](d5abadb)) * Retry request spot ([#3116](#3116)) ([82de5f1](82de5f1)) ### Miscellaneous * Add stdlib tests for pedersen commitment ([#3075](#3075)) ([87fa621](87fa621)) * Automatic c_binds for commit should return a point instead of an Fr element ([#3072](#3072)) ([2e289a5](2e289a5)) * Cleanup remaining mentions of `compress` with pedersen in cpp and ts ([#3074](#3074)) ([52cf383](52cf383)) * E2e on spots [ci rebuild] ([#3068](#3068)) ([15db6bf](15db6bf)) * Fix dapp_testing e2e race condition ([#3094](#3094)) ([89e7c21](89e7c21)) * Remove docs mirror ([#3122](#3122)) ([3fa51e2](3fa51e2)) * Remove endomorphism coefficient from ecc_add_gate ([#3115](#3115)) ([d294987](d294987)) * Remove unecessary calls to `pedersen__init` ([#3079](#3079)) ([84f8db2](84f8db2)) * Remove unused pedersen c_binds ([#3058](#3058)) ([e71e5f9](e71e5f9)) * Removes pedersen commit native pairs method ([#3073](#3073)) ([69a34c7](69a34c7)) * Rename private-kernel subpackage to protocol-circuits ([#3134](#3134)) ([3e07104](3e07104)) ### Documentation * Initial keys spec ([#3035](#3035)) ([4b24c58](4b24c58)) </details> <details><summary>barretenberg.js: 0.13.0</summary> ## [0.13.0](barretenberg.js-v0.12.0...barretenberg.js-v0.13.0) (2023-10-31) ### Features * New script to output table of benchmarks for README pasting. ([#2780](#2780)) ([6c20b45](6c20b45)) ### Miscellaneous * Automatic c_binds for commit should return a point instead of an Fr element ([#3072](#3072)) ([2e289a5](2e289a5)) * Remove unecessary calls to `pedersen__init` ([#3079](#3079)) ([84f8db2](84f8db2)) * Remove unused pedersen c_binds ([#3058](#3058)) ([e71e5f9](e71e5f9)) </details> <details><summary>barretenberg: 0.13.0</summary> ## [0.13.0](barretenberg-v0.12.0...barretenberg-v0.13.0) (2023-10-31) ### Features * Adding structure to Transcript ([#2937](#2937)) ([db67aa1](db67aa1)) * Efficient ZM quotient computation ([#3016](#3016)) ([ebda5fc](ebda5fc)) * Measure plonk rounds ([#3065](#3065)) ([c8e1d8b](c8e1d8b)) * New script to output table of benchmarks for README pasting. ([#2780](#2780)) ([6c20b45](6c20b45)) * Pedersen in typescript. ([#3111](#3111)) ([933f1b2](933f1b2)) * Protogalaxy folding of challenges ([#2935](#2935)) ([7ed30e8](7ed30e8)) * Zeromorph with concatenation (Goblin Translator part 10) ([#3006](#3006)) ([70b0f17](70b0f17)) ### Miscellaneous * Add stdlib tests for pedersen commitment ([#3075](#3075)) ([87fa621](87fa621)) * Automatic c_binds for commit should return a point instead of an Fr element ([#3072](#3072)) ([2e289a5](2e289a5)) * Cleanup remaining mentions of `compress` with pedersen in cpp and ts ([#3074](#3074)) ([52cf383](52cf383)) * Remove endomorphism coefficient from ecc_add_gate ([#3115](#3115)) ([d294987](d294987)) * Remove unecessary calls to `pedersen__init` ([#3079](#3079)) ([84f8db2](84f8db2)) * Remove unused pedersen c_binds ([#3058](#3058)) ([e71e5f9](e71e5f9)) * Removes pedersen commit native pairs method ([#3073](#3073)) ([69a34c7](69a34c7)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
🤖 I have created a release *beep* *boop* --- <details><summary>aztec-packages: 0.13.0</summary> ## [0.13.0](AztecProtocol/aztec-packages@aztec-packages-v0.12.0...aztec-packages-v0.13.0) (2023-10-31) ### ⚠ BREAKING CHANGES * PXE.getNotes(...) + refactor of note types ([#3051](AztecProtocol/aztec-packages#3051)) ### Features * `FieldNote` ([#3037](AztecProtocol/aztec-packages#3037)) ([3d1ffd0](AztecProtocol/aztec-packages@3d1ffd0)) * Add Aztec Boxes page to docs ([#2569](AztecProtocol/aztec-packages#2569)) ([997c15c](AztecProtocol/aztec-packages@997c15c)) * Adding structure to Transcript ([#2937](AztecProtocol/aztec-packages#2937)) ([db67aa1](AztecProtocol/aztec-packages@db67aa1)) * Compile noir contracts with noir_wasm ([#2737](AztecProtocol/aztec-packages#2737)) ([524cecf](AztecProtocol/aztec-packages@524cecf)) * Dockerize aztec-cli ([#3031](AztecProtocol/aztec-packages#3031)) ([ec2e3c2](AztecProtocol/aztec-packages@ec2e3c2)) * Efficient ZM quotient computation ([#3016](AztecProtocol/aztec-packages#3016)) ([ebda5fc](AztecProtocol/aztec-packages@ebda5fc)) * **feature_branch:** Private Kernel Circuit ([#2740](AztecProtocol/aztec-packages#2740)) ([f800a36](AztecProtocol/aztec-packages@f800a36)) * Measure plonk rounds ([#3065](AztecProtocol/aztec-packages#3065)) ([c8e1d8b](AztecProtocol/aztec-packages@c8e1d8b)) * Migrate the init kernel CPP tests to noir ([#3091](AztecProtocol/aztec-packages#3091)) ([906429f](AztecProtocol/aztec-packages@906429f)) * New script to output table of benchmarks for README pasting. ([#2780](AztecProtocol/aztec-packages#2780)) ([6c20b45](AztecProtocol/aztec-packages@6c20b45)) * Pedersen in typescript. ([#3111](AztecProtocol/aztec-packages#3111)) ([933f1b2](AztecProtocol/aztec-packages@933f1b2)) * Protogalaxy folding of challenges ([#2935](AztecProtocol/aztec-packages#2935)) ([7ed30e8](AztecProtocol/aztec-packages@7ed30e8)) * PXE.getNotes(...) + refactor of note types ([#3051](AztecProtocol/aztec-packages#3051)) ([16abb5a](AztecProtocol/aztec-packages@16abb5a)) * Zeromorph with concatenation (Goblin Translator part 10) ([#3006](AztecProtocol/aztec-packages#3006)) ([70b0f17](AztecProtocol/aztec-packages@70b0f17)) ### Bug Fixes * Bad contract txs publishing contract data ([#2673](AztecProtocol/aztec-packages#2673)) ([ccd4611](AztecProtocol/aztec-packages@ccd4611)) * Better error message for compute_note_hash_and_nullifier. ([#3097](AztecProtocol/aztec-packages#3097)) ([57bec53](AztecProtocol/aztec-packages@57bec53)) * Broken `FieldNote` test ([#3135](AztecProtocol/aztec-packages#3135)) ([fe78ecf](AztecProtocol/aztec-packages@fe78ecf)) * Docker-compose up, rather than run. ([#3081](AztecProtocol/aztec-packages#3081)) ([242f780](AztecProtocol/aztec-packages@242f780)) * Formatting ([#3070](AztecProtocol/aztec-packages#3070)) ([e1633d3](AztecProtocol/aztec-packages@e1633d3)) * Minor stale naming fix ([#3117](AztecProtocol/aztec-packages#3117)) ([a6786ae](AztecProtocol/aztec-packages@a6786ae)) * Push cli docker image to docker hub ([#3120](AztecProtocol/aztec-packages#3120)) ([ccad50f](AztecProtocol/aztec-packages@ccad50f)) * Remove duplicate terraform resource definition ([#3066](AztecProtocol/aztec-packages#3066)) ([d5abadb](AztecProtocol/aztec-packages@d5abadb)) * Retry request spot ([#3116](AztecProtocol/aztec-packages#3116)) ([82de5f1](AztecProtocol/aztec-packages@82de5f1)) ### Miscellaneous * Add stdlib tests for pedersen commitment ([#3075](AztecProtocol/aztec-packages#3075)) ([87fa621](AztecProtocol/aztec-packages@87fa621)) * Automatic c_binds for commit should return a point instead of an Fr element ([#3072](AztecProtocol/aztec-packages#3072)) ([2e289a5](AztecProtocol/aztec-packages@2e289a5)) * Cleanup remaining mentions of `compress` with pedersen in cpp and ts ([#3074](AztecProtocol/aztec-packages#3074)) ([52cf383](AztecProtocol/aztec-packages@52cf383)) * E2e on spots [ci rebuild] ([#3068](AztecProtocol/aztec-packages#3068)) ([15db6bf](AztecProtocol/aztec-packages@15db6bf)) * Fix dapp_testing e2e race condition ([#3094](AztecProtocol/aztec-packages#3094)) ([89e7c21](AztecProtocol/aztec-packages@89e7c21)) * Remove docs mirror ([#3122](AztecProtocol/aztec-packages#3122)) ([3fa51e2](AztecProtocol/aztec-packages@3fa51e2)) * Remove endomorphism coefficient from ecc_add_gate ([#3115](AztecProtocol/aztec-packages#3115)) ([d294987](AztecProtocol/aztec-packages@d294987)) * Remove unecessary calls to `pedersen__init` ([#3079](AztecProtocol/aztec-packages#3079)) ([84f8db2](AztecProtocol/aztec-packages@84f8db2)) * Remove unused pedersen c_binds ([#3058](AztecProtocol/aztec-packages#3058)) ([e71e5f9](AztecProtocol/aztec-packages@e71e5f9)) * Removes pedersen commit native pairs method ([#3073](AztecProtocol/aztec-packages#3073)) ([69a34c7](AztecProtocol/aztec-packages@69a34c7)) * Rename private-kernel subpackage to protocol-circuits ([#3134](AztecProtocol/aztec-packages#3134)) ([3e07104](AztecProtocol/aztec-packages@3e07104)) ### Documentation * Initial keys spec ([#3035](AztecProtocol/aztec-packages#3035)) ([4b24c58](AztecProtocol/aztec-packages@4b24c58)) </details> <details><summary>barretenberg.js: 0.13.0</summary> ## [0.13.0](AztecProtocol/aztec-packages@barretenberg.js-v0.12.0...barretenberg.js-v0.13.0) (2023-10-31) ### Features * New script to output table of benchmarks for README pasting. ([#2780](AztecProtocol/aztec-packages#2780)) ([6c20b45](AztecProtocol/aztec-packages@6c20b45)) ### Miscellaneous * Automatic c_binds for commit should return a point instead of an Fr element ([#3072](AztecProtocol/aztec-packages#3072)) ([2e289a5](AztecProtocol/aztec-packages@2e289a5)) * Remove unecessary calls to `pedersen__init` ([#3079](AztecProtocol/aztec-packages#3079)) ([84f8db2](AztecProtocol/aztec-packages@84f8db2)) * Remove unused pedersen c_binds ([#3058](AztecProtocol/aztec-packages#3058)) ([e71e5f9](AztecProtocol/aztec-packages@e71e5f9)) </details> <details><summary>barretenberg: 0.13.0</summary> ## [0.13.0](AztecProtocol/aztec-packages@barretenberg-v0.12.0...barretenberg-v0.13.0) (2023-10-31) ### Features * Adding structure to Transcript ([#2937](AztecProtocol/aztec-packages#2937)) ([db67aa1](AztecProtocol/aztec-packages@db67aa1)) * Efficient ZM quotient computation ([#3016](AztecProtocol/aztec-packages#3016)) ([ebda5fc](AztecProtocol/aztec-packages@ebda5fc)) * Measure plonk rounds ([#3065](AztecProtocol/aztec-packages#3065)) ([c8e1d8b](AztecProtocol/aztec-packages@c8e1d8b)) * New script to output table of benchmarks for README pasting. ([#2780](AztecProtocol/aztec-packages#2780)) ([6c20b45](AztecProtocol/aztec-packages@6c20b45)) * Pedersen in typescript. ([#3111](AztecProtocol/aztec-packages#3111)) ([933f1b2](AztecProtocol/aztec-packages@933f1b2)) * Protogalaxy folding of challenges ([#2935](AztecProtocol/aztec-packages#2935)) ([7ed30e8](AztecProtocol/aztec-packages@7ed30e8)) * Zeromorph with concatenation (Goblin Translator part 10) ([#3006](AztecProtocol/aztec-packages#3006)) ([70b0f17](AztecProtocol/aztec-packages@70b0f17)) ### Miscellaneous * Add stdlib tests for pedersen commitment ([#3075](AztecProtocol/aztec-packages#3075)) ([87fa621](AztecProtocol/aztec-packages@87fa621)) * Automatic c_binds for commit should return a point instead of an Fr element ([#3072](AztecProtocol/aztec-packages#3072)) ([2e289a5](AztecProtocol/aztec-packages@2e289a5)) * Cleanup remaining mentions of `compress` with pedersen in cpp and ts ([#3074](AztecProtocol/aztec-packages#3074)) ([52cf383](AztecProtocol/aztec-packages@52cf383)) * Remove endomorphism coefficient from ecc_add_gate ([#3115](AztecProtocol/aztec-packages#3115)) ([d294987](AztecProtocol/aztec-packages@d294987)) * Remove unecessary calls to `pedersen__init` ([#3079](AztecProtocol/aztec-packages#3079)) ([84f8db2](AztecProtocol/aztec-packages@84f8db2)) * Remove unused pedersen c_binds ([#3058](AztecProtocol/aztec-packages#3058)) ([e71e5f9](AztecProtocol/aztec-packages@e71e5f9)) * Removes pedersen commit native pairs method ([#3073](AztecProtocol/aztec-packages#3073)) ([69a34c7](AztecProtocol/aztec-packages@69a34c7)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
We need to fold challenges as well. This means a refactor around how relation parameters are specified and relation degrees are calculated.
The
RelationParametersclass is now templated on a generic typeTthat could be a field or could be aUnivariateclass.We need separate terminology for the length (==degree + 1) of a polynomial depending on whether we regard the challenges as constants or variables. I settled on "partial" and "total" since I think "partial degree" and "total degree" is what you'd expect in math. The way we calculate the total degree is to add some adjust ment factors. We only had to add these in three relations (the only three that make use of relation parameters). NB I focused only on relations for flavors where we intend to do folding.
If$f$ has length $l_f$ and $g$ has length $l_g$ , then $f(g(X))$ has degree $(l_f-1)\cdot (l_g-1)$ , hence its length is $(l_f - 1) \cdot (l_g - 1) + 1$ . We need to calculate these of this form in cases now where the challenge parameters in $f$ are regarded as variable.
Since I had to refactor the relations, I do some light touching up of the relations for uniformity. I also move the Goblin Translator relations into a folder (like we already had for ECCVM) and move some of the Goblin relations that were contained in other relation files out into their own files.
I made some comments en route to calculating the total degree of the auxiliary relation. I leave those in since I think they'll be helpful later.
Checklist:
Remove the checklist to signal you've completed it. Enable auto-merge if the PR is ready to merge.