-
Notifications
You must be signed in to change notification settings - Fork 600
feat: multiPCS infra for Ultra and Mega #20230
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
iakovenkos
wants to merge
62
commits into
next
Choose a base branch
from
si/multipcs-proto
base: next
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
62 commits
Select commit
Hold shift + click to select a range
d6ddd76
add analysis
iakovenkos 2b87428
add interleaved pippenger
iakovenkos 9ef3957
add benches
iakovenkos 351f0e5
first approximation
iakovenkos 6dd3845
next step smth
iakovenkos 6e1a58e
manifests match
iakovenkos 6f3b684
some ordering fixes
iakovenkos 54c0cb2
add multipcs tests and polynomial shift support
iakovenkos 9a04119
rm infos
iakovenkos 974af8d
debugging
iakovenkos 2c379d2
tests ok, cleanup
iakovenkos f81f69b
clean-up
iakovenkos 1131d22
MultiMega ZK and recursive verification working
iakovenkos 544c9e0
Enable REPEATED_COMMITMENTS optimization for MultiMega
iakovenkos 710c642
bench + upd md
iakovenkos 1ccbab0
Merge remote-tracking branch 'origin/merge-train/barretenberg' into s…
iakovenkos e67028f
Merge branch 'merge-train/barretenberg' into si/multipcs-proto
iakovenkos 6eabcf3
delete stale mds
iakovenkos 5a0d701
fixes after bad merge
iakovenkos 27544b2
Merge remote-tracking branch 'origin/merge-train/barretenberg' into s…
iakovenkos cd36385
backport
iakovenkos ac88439
rm clutter
iakovenkos 6e505f1
Reduce MultiHonk duplication via inheritance and rename MultiMega -> …
iakovenkos 613ade1
update tests
iakovenkos 438c664
fix tests
iakovenkos 06335d7
deduplicating
iakovenkos b2e7838
delete multi files
iakovenkos 5f2cac8
delete multichonk md
iakovenkos 8b77834
rm verbose shplemini interleaving tests, clean up verifier instance
iakovenkos e68d10f
unify further
iakovenkos 1fedc22
Merge remote-tracking branch 'origin/merge-train/barretenberg' into s…
iakovenkos 64ce4b3
fix a couple of merge issues
iakovenkos d1e52b8
prover and verifier are almost agnostic to the batch size
iakovenkos 56ef0bc
rm branching from oink verifier
iakovenkos 0f5257d
tiny clean up
iakovenkos 08d9329
fix batched translator test# Please enter the commit message for your…
iakovenkos 8e74526
Merge remote-tracking branch 'origin/merge-train/barretenberg' into s…
iakovenkos e553d05
fix merge conflict resolutions: adopt HasGeminiMasking, update batche…
iakovenkos 5d82742
fix build/merge issues
iakovenkos 218db6a
test fixes
iakovenkos 4b759ac
mem efficient interleaving
iakovenkos 87acc7f
fix megahonk<4>
iakovenkos ed0ee73
decluttering
iakovenkos 866ae66
rm dead code
iakovenkos 49f1c1e
moore clean up
iakovenkos f131a2f
Merge remote-tracking branch 'origin/merge-train/barretenberg' into s…
iakovenkos b77cfe6
clean up
iakovenkos 723ada8
fixes and deduplication
iakovenkos cca146f
tests green
iakovenkos b7c0124
format
iakovenkos 542de10
rm clutter from prover polys
iakovenkos 30de600
deduplicate proof length
iakovenkos 7aacfcb
fix tests
iakovenkos 6a33240
Merge branch 'merge-train/barretenberg' into si/multipcs-proto
iakovenkos 47be4a8
fix docs
iakovenkos cb2384e
memory-efficient on-the-fly interleaving
iakovenkos d3d63e5
simplify
iakovenkos 49dfb34
unified BS-agnostic oink prover and verifier
iakovenkos 6b9a7e8
add DualMega (BS=2) interleaving flavor
iakovenkos 841aa7c
add DualUltra (BS=2) interleaving flavor
iakovenkos a6298a7
rm shpl bench
iakovenkos 43ae4ec
review fixes and enable DualUltra/DualUltraZK in UltraHonk tests
iakovenkos File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -57,32 +57,48 @@ template <typename Curve> class BatchedHonkTranslatorVerifier_ { | |
| using TransBF = typename TransFlavor::BF; | ||
|
|
||
| // Joint RepeatedCommitmentsData for Shplemini's remove_repeated_commitments optimization. | ||
| // Joint unshifted = [Trans_unshifted(TU), MZK_unshifted(P+W)]. The translator's gemini_masking_poly | ||
| // is at position 0 of unshifted and is consumed by Shplemini's offset=2 (Q + masking). | ||
| // After Shplemini's offset=2, the virtual layout is: | ||
| // Unshifted: [Trans_rest(TU-1) | MZK_precomputed(P) | MZK_witness(W)] | ||
| // Shifted: [MZK_shifted(S) | Trans_shifted(TS)] | ||
| // After Shplemini's offset=2 (Q_commitment + translator_masking_poly), the virtual layout is: | ||
| // Unshifted: [Trans_unshifted_no_masking(TU-1) | MegaZK_precomputed(P) | MegaZK_witness(W)] | ||
| // Shifted: [MegaZK_shifted(S) | Trans_shifted(TS)] | ||
| // | ||
| // Range 1 (Translator merged): ordered(5)+z_perm(1)+concat(5) in unshifted ↔ same in shifted | ||
| // Range 2 (MegaZK): witness[0..S-1] ↔ mega_zk_shifted[0..S-1] | ||
| // Range 1 (Translator): ordered(5)+z_perm(1) in unshifted ↔ same in shifted | ||
| // Range 2 (Translator): concat(5) in unshifted ↔ same in shifted | ||
| // Combined into two DuplicateRanges. | ||
| static constexpr RepeatedCommitmentsData REPEATED_COMMITMENTS = [] { | ||
| constexpr size_t TU = TranslatorFlavor::NUM_PCS_UNSHIFTED; // includes masking(1) | ||
| constexpr size_t P = MegaZKFlavorT::NUM_PRECOMPUTED_ENTITIES; | ||
| constexpr size_t W = MegaZKFlavorT::NUM_WITNESS_ENTITIES; | ||
| constexpr size_t W = MegaZKFlavorT::REPEATED_COMMITMENTS.first.duplicate_start - | ||
| MegaZKFlavorT::REPEATED_COMMITMENTS.first.original_start; | ||
| constexpr size_t S = MegaZKFlavorT::NUM_SHIFTED_ENTITIES; | ||
| // Translator repeated: ordered(5)+z_perm(1)+concat(5) in Trans_rest ↔ Trans_shifted | ||
| // Trans_rest starts at virtual 0; repeated starts at ordered_extra(1)+op(1)=2 | ||
| constexpr size_t TRANS_REPEAT_START = TranslatorFlavor::REPEATED_COMMITMENTS.first.original_start; | ||
| constexpr size_t TRANS_REPEAT_COUNT = | ||
| TranslatorFlavor::REPEATED_COMMITMENTS.first.count + TranslatorFlavor::REPEATED_COMMITMENTS.second.count; | ||
| // In shifted section: op_queue entries precede the repeated entries | ||
| constexpr size_t TRANS_SHIFTED_SKIP = TranslatorFlavor::NUM_PCS_TO_BE_SHIFTED - TRANS_REPEAT_COUNT; | ||
| return RepeatedCommitmentsData(TRANS_REPEAT_START, // Translator original in unshifted | ||
| (TU - 1) + P + W + S + TRANS_SHIFTED_SKIP, // Translator duplicate in shifted | ||
| TRANS_REPEAT_COUNT, // Translator count | ||
| (TU - 1) + P, // MegaZK original: witness start in unshifted | ||
| (TU - 1) + P + W, // MegaZK duplicate: shifted start | ||
| S); // MegaZK count | ||
| constexpr size_t TU = TranslatorFlavor::NUM_PCS_UNSHIFTED; | ||
| constexpr size_t TS = TranslatorFlavor::NUM_PCS_TO_BE_SHIFTED; | ||
| // In the joint prover poly array (after offset=2 consumes Q + trans masking): | ||
| // Unshifted: [ordered_extra(1), op(1), ordered(5), z_perm(1), concat(5), mega_precomputed(P), mega_witness(W)] | ||
| // Shifted: [mega_shifted(S), op_queue(3), ordered(5), z_perm(1), concat(5)] | ||
| // | ||
| // Translator repeated: | ||
| // Range 1: ordered(5)+z_perm(1) at unshifted[2..7] ↔ shifted[(TU-1)+P+W+S+3..(TU-1)+P+W+S+8] | ||
| // Range 2: concat(5) at unshifted[8..12] ↔ shifted[(TU-1)+P+W+S+9..(TU-1)+P+W+S+13] | ||
| // MegaZK repeated: | ||
| // witness ↔ mega_shifted: unshifted[(TU-1)+P-1..] ↔ shifted[(TU-1)+P+W-1..] | ||
| // (using MegaZK standalone offsets P-1, P+W-1 shifted by TU-1) | ||
| // | ||
| // We use two DuplicateRanges. Range 1 = translator ordered+z_perm+concat, Range 2 = MegaZK. | ||
| constexpr size_t TRANS_ORIG_START = | ||
| TranslatorFlavor::REPEATED_COMMITMENTS.first.original_start; // 2 (ordered[0]) | ||
| constexpr size_t TRANS_TOTAL_COUNT = TranslatorFlavor::REPEATED_COMMITMENTS.first.count + | ||
| TranslatorFlavor::REPEATED_COMMITMENTS.second.count; // 6+5=11 | ||
| // op_queue skip in shifted = 3 | ||
| constexpr size_t TRANS_SHIFTED_SKIP = TS - TRANS_TOTAL_COUNT; // 14-11 = 3 (op_queue wires are not repeated) | ||
| constexpr size_t MEGA_ZK_ORIG = (TU - 1) + MegaZKFlavorT::REPEATED_COMMITMENTS.first.original_start; | ||
| constexpr size_t MEGA_ZK_DUP = (TU - 1) + MegaZKFlavorT::REPEATED_COMMITMENTS.first.duplicate_start; | ||
| return RepeatedCommitmentsData(TRANS_ORIG_START, // Trans original: ordered[0] in unshifted | ||
| (TU - 1) + P + W + S + | ||
| TRANS_SHIFTED_SKIP, // Trans duplicate: ordered[0] in shifted | ||
| TRANS_TOTAL_COUNT, // Trans count: 11 | ||
| MEGA_ZK_ORIG, // MegaZK original: witness start in unshifted | ||
| MEGA_ZK_DUP, // MegaZK duplicate: mega_shifted start in shifted | ||
| S, // MegaZK count | ||
| 2 /* shplemini_offset: Q + translator_masking_poly */); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Where is the 2 coming from? |
||
| }(); | ||
|
|
||
| /** | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -35,7 +35,8 @@ template <typename Curve> struct ClaimBatcher_ { | |
| }; | ||
|
|
||
| std::optional<Batch> unshifted; // commitments and evaluations of unshifted polynomials | ||
| std::optional<Batch> shifted; // commitments of to-be-shifted-by-1 polys, evals of their shifts | ||
| std::optional<Batch> shifted; // commitments of to-be-shifted polys, evals of their shifts | ||
| size_t shift_exponent = 1; // shift depth: 1 for standard (G/X), k for interleaved (G/X^k) | ||
|
|
||
| Batch get_unshifted() { return (unshifted) ? *unshifted : Batch{}; } | ||
| Batch get_shifted() { return (shifted) ? *shifted : Batch{}; } | ||
|
|
@@ -47,9 +48,10 @@ template <typename Curve> struct ClaimBatcher_ { | |
| * @details Computes scalars s_0, s_1 given by | ||
| * \f[ | ||
| * - s_0 = \left(\frac{1}{z-r} + \nu \times \frac{1}{z+r}\right) \f], | ||
| * - s_1 = \frac{1}{r} \times \left(\frac{1}{z-r} - \nu \times \frac{1}{z+r}\right) | ||
| * - s_1 = \frac{1}{r^k} \times \left(\frac{1}{z-r} - \nu \times \frac{1}{z+r}\right) | ||
| * \f] | ||
| * where the scalars used to batch the claims are given by | ||
| * where k is the shift_exponent member (1 for standard shifts, BS for interleaved polynomials), | ||
| * and the scalars used to batch the claims are given by | ||
| * \f[ | ||
| * \left( | ||
| * - s_0, | ||
|
|
@@ -77,9 +79,25 @@ template <typename Curve> struct ClaimBatcher_ { | |
| unshifted->scalar = inverse_vanishing_eval_pos + nu_challenge * inverse_vanishing_eval_neg; | ||
| } | ||
| if (shifted) { | ||
| // r⁻¹ ⋅ (1/(z−r) − ν/(z+r)) | ||
| shifted->scalar = | ||
| r_challenge.invert() * (inverse_vanishing_eval_pos - nu_challenge * inverse_vanishing_eval_neg); | ||
| // r⁻ᵏ ⋅ (1/(z−r) + (-1)^k ⋅ ν/(z+r)) where k is the shift_exponent | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe it's worth giving some more details here, like why is it (-1)^k and not always -1 |
||
| // This comes from A₀₋(X) = F(X) + (-1)^k · G(X)/r^k, needed because (-r)^k = (-1)^k · r^k | ||
| // For standard shifts k=1 (odd): r⁻¹ ⋅ (1/(z−r) − ν/(z+r)) | ||
| // For interleaved shifts k=4 (even): r⁻⁴ ⋅ (1/(z−r) + ν/(z+r)) | ||
| if (shift_exponent == 1) { | ||
| // Fast path: avoid extra multiplication by neg_sign (important for recursive verifiers) | ||
| shifted->scalar = | ||
| r_challenge.invert() * (inverse_vanishing_eval_pos - nu_challenge * inverse_vanishing_eval_neg); | ||
| } else { | ||
| Fr r_power = r_challenge; | ||
| for (size_t i = 1; i < shift_exponent; ++i) { | ||
| r_power *= r_challenge; | ||
| } | ||
| const Fr r_inv_shift = r_power.invert(); | ||
| // (-1)^k: even k gives +1, odd k gives -1 (but k=1 handled above) | ||
| const Fr neg_sign = (shift_exponent % 2 == 0) ? Fr(1) : Fr(-1); | ||
| shifted->scalar = | ||
| r_inv_shift * (inverse_vanishing_eval_pos + neg_sign * nu_challenge * inverse_vanishing_eval_neg); | ||
| } | ||
| } | ||
| } | ||
| /** | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
WHy was this not needed before?