Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streamline macro syntax #169

Merged
merged 6 commits into from
Dec 9, 2023
Merged

Streamline macro syntax #169

merged 6 commits into from
Dec 9, 2023

Conversation

huitseeker
Copy link
Contributor

@huitseeker huitseeker commented Dec 9, 2023

The Problem

The current macro system requires learning what 7 auxiliary macros do: zip_with, zip_with_for_each, zip_with_into_par_iter, zip_with_iter, zip_with_par_iter, zip_with_par_iter_mut_for_each, zip_with_into_iter.

This PR

Switches us to a mode in which the user has only 2 macros to learn about : zip_with and zip_with_for_each.
This further documents them as such:

// no iterator projection specified: the macro assumes the arguments *are* iterators
// zip_with!((iter1, iter2, iter3), |a, b, c| a + b + c) -> iter1.zip_eq(iter2.zip_eq(iter3)).map(|(a, (b, c))| a + b + c)
//
// iterator projection specified: use it on each argument
// zip_with!(par_iter, (vec1, vec2, vec3), |a, b, c| a + b + c) -> 
// vec1.par_iter().zip_eq(vec2.par_iter().zip_eq(vec3.par_iter())).map(|(a, (b, c))| a + b + c)

and:

// no iterator projection specified: the macro assumes the arguments *are* iterators
// zip_with_for_each!((iter1, iter2, iter3), |a, b, c| a + b + c) -> iter1.zip_eq(iter2.zip_eq(iter3)).for_each(|(a, (b, c))| a + b + c)
//
// iterator projection specified: use it on each argument
// zip_with_for_each!(par_iter, (vec1, vec2, vec3), |a, b, c| a + b + c) ->
//   vec1.par_iter().zip_eq(vec2.par_iter().zip_eq(vec3.par_iter())).for_each(|(a, (b, c))| a + b + c)

Everything is refactored to use one of these two macros. While there are more complex forms to these macros, the user doesn't need to import them or know about them. The implementations are also simplified.

Details

Also contains #170 because stacked PRs in Git are a mess.

Details:
- Simplified the implementation of `BatchedRelaxedR1CSSNARKTrait` by replacing usage of `zip_with_iter` and related macros with `zip_with_fn`.
- Eliminated `zip_with_iter`, `zip_with_par_iter`, `zip_with_into_iter`, and `zip_with_into_par_iter` macros from the codebase, reducing repetition and enhancing code organization.
@huitseeker huitseeker requested a review from mpenciak December 9, 2023 14:26
@huitseeker huitseeker force-pushed the streamlining branch 3 times, most recently from ba22019 to 3189aee Compare December 9, 2023 14:41
- Replaced the `zip_with_fn!` function with the `zip_with!` function in various files,
- Refactored `zip_with` and `zip_with_for_each` macros in `src/spartan/macros.rs` for improved handling of iterator projection specification.
- Consolidated `zip_all_with_fn` macro's functionality into `zip_all` and eliminated unused macros.
@huitseeker huitseeker requested a review from adr1anh December 9, 2023 17:37
@huitseeker huitseeker mentioned this pull request Dec 9, 2023
porcuquine
porcuquine previously approved these changes Dec 9, 2023
Copy link
Contributor

@porcuquine porcuquine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a natural next (perhaps final) evolution of these macros.

* refactor: removed needless copies of `R1CSShape`

- Modified the `primary_r1cs_shapes` function in `supernova/mod.rs` to return a vector of references, improving memory efficiency.
- Use those `primary_r1cs_shapes` in the `setup` and `prove` methods of `CompressedSNARK`.
- Altered the `BatchedRelaxedR1CSSNARKTrait` trait functions in `traits/snark.rs` to accept vector of references instead of reference,

* refactor: Refactor polynomial size assertion checks in sumcheck.rs

- Refactors the length assertion checks in the `prove_quad_batch` and `prove_cubic_with_additive_term_batch` methods in `src/spartan/sumcheck.rs`.
- Simplified the code by replacing multiple `assert_eq` calls with a loop.
- Removed the "nested_tuple" macro from the "src/spartan/macros.rs" file.
@huitseeker huitseeker added this pull request to the merge queue Dec 9, 2023
Merged via the queue into batched_spartan with commit ad4867b Dec 9, 2023
3 checks passed
@huitseeker huitseeker deleted the streamlining branch December 9, 2023 19:27
github-merge-queue bot pushed a commit that referenced this pull request Dec 11, 2023
* skeleton of what is to be done

* wip inner and outer sumchecks done

* fix borrow checker issues

* batched spartan prove done

* Small improvements
- Irregular shape yields error
- Remove (W,E) clones
- Use Math module for log2
- Generate tau using only 1 challenge
- Remove some `clone()`
- Comments

* Fix evaluation points

* Implement verifier

* `CompressedSNARK` implementation

* Tests fixed, failing

* fix sumcheck, test passes

* bound -> bind

* remove todo

* add test

* Document BatchedRelaxedR1CSSNARKTrait and SuperNova CompressedSNARK.

* working

* - Remove debug code
- Add Sumcheck::verify_batch
- Uniformize batching of evals
- Comments

* Adress review comments
- add itertools for multiunzip
- avoid clones
- avoid recomputing sizes for asserts

* Fix asserts

* batched ppsnark

* Fix scaling

* - Fix padding of poly_W
- Optimized evaluation
- use batch_diff_size
- use multiunzip

* rebase fixes

* Only create taus once.

* Avoid clone by not allocating PowPolynomial.

* Riff on review.

* refactor: Implement `.zip_eq()` for equal length iterator safety

- Replaced `.zip()` with `.zip_eq()` across various files including `sumcheck.rs`, `mod.rs`, `batched_ppsnark.rs`, `batched.rs` and `snark.rs`, ensuring equal length enforcement in iterations and reducing out-of-bound errors.
- Updated a multiunzip resulting in a double iteration in batched_ppsnark::prove.

* add benchmark

* fix witnes

* fix: clean up iterators

* feat: Introduce new utility macros in spartan module

- Introduced a macro `zip_with` in `src/spartan/mod.rs` to simplify the process of zipping multiple iterators and applying a single parallel function.
- Developed additional macros `nested_tuple` and `nested_idents` for detailed nested tuple patterns and identifiers respectively.
- Implemented `zip_all` macro that acts as a fold-right zipping operator for all given expressions.

* refactor: Refactor batched.rs to enhance code readability

- Refactored `src/spartan/batched.rs` to use `zip_with()` macro improving the readability, simplification, and consistent usage in the code.
- Inserted comments highlighting sections marked for review or possible future code simplification.

* refactor: use zip_with! in batch_ppsnark

- Updated `batched_ppsnark.rs` in the `src/spartan` directory with uses of zip_with macro

* ignore local macro doctest

* Use zip_with.

* Implement and use zip_with_for_each.

* Dubious double zip_with.

* More zip_with.

* Small fixes
- `zip_with` for `batched_ppsnark`
- Complete comment of `BatchedRelaxedR1CSSNARK`
- Restore Abomonation bounds
- Fix nits in `multilinear.rs`

* refactor: more zip_with instances (#153)

* refactor: more instances of zip_with

* Supernova: Some rewrites (#154)

* refactor: more zip_with

* feat: Implement `FromIterator` for polynomial types

- Enhanced `EqPolynomial`, `MultilinearPolynomial`, and `SparsePolynomial` with `FromIterator` implementation.

* refactor: use instaces of Polynomial Fromiterator where possible

* refactor: remove unused impls of FromIterator

* refactor: address review comments

* refactor: enhance scope visibility scope in Spartan and Supernova modules (#156)

Absolute visibility modifiers tend to not inadvertently make things public when you move code, contrarily to (super).

- Changed the visibility scope of several fields and methods within multiple substrates in the `ppsnark.rs` file to be within the `crate::spartan`.
- Altered the module import path in the `circuit.rs` file from `super` to `crate::supernova`.
- Modified the visibility of the `WitnessBoundSumcheck` in `batched_ppsnark.rs`, the visibility scope is now set to `crate::spartan`.
- Scoped `batch_eval_prove` and `batch_eval_verify` functions within `crate::spartan` in `snark.rs`.

* Avoid EqPolynomial when only for evals.

* refactor: a few more instances of EqPolynomial::evals_from_points

- in `multilinear.rs` and `ipa_pc.rs`, replaces the previous method `EqPolynomial::new(r.to_vec()).evals()` with a more efficient `EqPolynomial::evals_from_points(r)`.

* doc: fix a few nits (#160)

* refactor: rename powers -> squares

- Renamed the `powers` method to `squares` in `PowPolynomial` struct within `src/spartan/polys/power.rs` and changed its visibility level.

* fix: comment

* fix: comment of batch

* derive fixes

* fix: compressed snark benchmark

- Introduced `compressed-snark-supernova` within the `Cargo.toml` file.
- fixed Rust

* ci: Add trigger for PRs to batched_spartan (#165)

* More zip with (#158)

* Expand zip_with macros to accommodate more use.

* Clippy.

* Move Spartan macros to own module.

* Only use zip_with_fn to implement convenience macros.

* Use convenience zip_with macros more.

* Remove for_each variants of zip_with macros.

* Remove flat_map variants of zip_with.

---------

Co-authored-by: porcuquine <[email protected]>

* Introduce proper `MaskedEq` poly (#162)

* Introduce proper `MaskedEq` poly

* refactor: Refactor MaskedEq poly for improved testing and performance

- Introduce lifetime parameters to `MaskedEqPolynomial` struct to improve memory management.
- Modify the `MaskedEqPolynomial` creation process so it now requires a reference to an existing `EqPolynomial`.
- Adapt corresponding tests to reflect the above changes.

* Fix fmt & derive debug

---------

Co-authored-by: François Garillot <[email protected]>

* Restore zip_with_for_each. (#166)

Co-authored-by: porcuquine <[email protected]>

* fix benchmark (#168)

* Sumcheck update (#163)

* Include number of instances in transcript

* Clarify Sumcheck and verifier

* Fix clippy

* Streamline macro syntax (#169)

* refactor: Refactor Spartan code for streamlined use of `zip_with_fn`

Details:
- Simplified the implementation of `BatchedRelaxedR1CSSNARKTrait` by replacing usage of `zip_with_iter` and related macros with `zip_with_fn`.
- Eliminated `zip_with_iter`, `zip_with_par_iter`, `zip_with_into_iter`, and `zip_with_into_par_iter` macros from the codebase, reducing repetition and enhancing code organization.

* refactor: Refactor zip functions

- Replaced the `zip_with_fn!` function with the `zip_with!` function in various files,
- Refactored `zip_with` and `zip_with_for_each` macros in `src/spartan/macros.rs` for improved handling of iterator projection specification.
- Consolidated `zip_all_with_fn` macro's functionality into `zip_all` and eliminated unused macros.

* fix: remove brackets

* Small fixes (#170)

* refactor: removed needless copies of `R1CSShape`

- Modified the `primary_r1cs_shapes` function in `supernova/mod.rs` to return a vector of references, improving memory efficiency.
- Use those `primary_r1cs_shapes` in the `setup` and `prove` methods of `CompressedSNARK`.
- Altered the `BatchedRelaxedR1CSSNARKTrait` trait functions in `traits/snark.rs` to accept vector of references instead of reference,

* refactor: Refactor polynomial size assertion checks in sumcheck.rs

- Refactors the length assertion checks in the `prove_quad_batch` and `prove_cubic_with_additive_term_batch` methods in `src/spartan/sumcheck.rs`.
- Simplified the code by replacing multiple `assert_eq` calls with a loop.

* refactor: Remove "nested_tuple" macro from spartan macros

- Removed the "nested_tuple" macro from the "src/spartan/macros.rs" file.

* refactor: pedantic calls in tests

---------

Co-authored-by: Matej Penciak <[email protected]>
Co-authored-by: Adrian Hamelink <[email protected]>
Co-authored-by: porcuquine <[email protected]>
Co-authored-by: Hanting Zhang <[email protected]>
Co-authored-by: Samuel Burnham <[email protected]>
Co-authored-by: porcuquine <[email protected]>
Co-authored-by: Adrian Hamelink <[email protected]>
Co-authored-by: Matej Penciak <[email protected]>
huitseeker added a commit to huitseeker/Nova that referenced this pull request Jan 10, 2024
- Created a new file `macros.rs` with the implementation of `zip_with` and `zip_with_for_each` macros providing syntactic sugar for zipWith patterns.
- zipWith patterns implemented through the use of the zip_with! macros now resolve to the use of zip_eq, a variant of zip that panics when the iterator arguments are of different length,
- the zip_eq implementation is the native rayon one for parallel iterators, and the one from itertools (see below) for the sequential ones,
- Optimized and refactored functions like `batch_eval_prove` and `batch_eval_verify` in `snark.rs`, methods inside `PolyEvalWitness` and `PolyEvalInstance` in `mod.rs`, and multiple functions in `multilinear.rs` through the use of implemented macros.

- Introduced the use of itertools::Itertools in various files to import the use of zip_eq on sequential iterators.
- Made use of the Itertools library for refactoring and optimizing computation in `sumcheck.rs` and `eq.rs` files.

This backports (among others) content from the following Arecibo PRS:
- lurk-lang/arecibo#149
- lurk-lang/arecibo#158
- lurk-lang/arecibo#169

Co-authored-by: porcuquine <[email protected]>
srinathsetty pushed a commit to microsoft/Nova that referenced this pull request Jan 10, 2024
- Created a new file `macros.rs` with the implementation of `zip_with` and `zip_with_for_each` macros providing syntactic sugar for zipWith patterns.
- zipWith patterns implemented through the use of the zip_with! macros now resolve to the use of zip_eq, a variant of zip that panics when the iterator arguments are of different length,
- the zip_eq implementation is the native rayon one for parallel iterators, and the one from itertools (see below) for the sequential ones,
- Optimized and refactored functions like `batch_eval_prove` and `batch_eval_verify` in `snark.rs`, methods inside `PolyEvalWitness` and `PolyEvalInstance` in `mod.rs`, and multiple functions in `multilinear.rs` through the use of implemented macros.

- Introduced the use of itertools::Itertools in various files to import the use of zip_eq on sequential iterators.
- Made use of the Itertools library for refactoring and optimizing computation in `sumcheck.rs` and `eq.rs` files.

This backports (among others) content from the following Arecibo PRS:
- lurk-lang/arecibo#149
- lurk-lang/arecibo#158
- lurk-lang/arecibo#169

Co-authored-by: porcuquine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants